I don't know what could be done to help with those problems. Here is what the goroutin

Health check crashes while printing a log statement about rain HOT 5 CLOSED

cenkalti commented on May 28, 2024

Health check crashes while printing a log statement

from rain.

Comments (5)

cenkalti commented on May 28, 2024

The health check mechanism is there for finding deadlocks in torrent loop and failing fast.

rain/torrent/session_healthcheck.go

Lines 9 to 12 in 3b0d35e

 // checkTorrent pings the torrent run loop periodically and crashes the program if a torrent does not respond in 

 // specified timeout. This is not a good behavior for a production program but it helps to find deadlocks easily, 

 // at least while developing. 

 func (s *Session) checkTorrent(t *torrent) {

The loop in https://github.com/cenkalti/rain/blob/master/torrent/torrent_run.go must not ever block at any time. All IO operations is done in separate goroutines that send their result to the select block in torrent_run.go using channels.

When I look at the traceback you provided, I see this:

[air:Desktop]% cat rain-crash.txt| grep torrent_run.go
	/var/lib/jenkins/go/pkg/mod/github.com/bsergean/[email protected]/torrent/torrent_run.go:20 +0x7ca
	/var/lib/jenkins/go/pkg/mod/github.com/bsergean/[email protected]/torrent/torrent_run.go:20 +0x7ca
	/var/lib/jenkins/go/pkg/mod/github.com/bsergean/[email protected]/torrent/torrent_run.go:20 +0x7ca
	/var/lib/jenkins/go/pkg/mod/github.com/bsergean/[email protected]/torrent/torrent_run.go:86 +0xf85
	/var/lib/jenkins/go/pkg/mod/github.com/bsergean/[email protected]/torrent/torrent_run.go:20 +0x7ca
	/var/lib/jenkins/go/pkg/mod/github.com/bsergean/[email protected]/torrent/torrent_run.go:20 +0x7ca
	/var/lib/jenkins/go/pkg/mod/github.com/bsergean/[email protected]/torrent/torrent_run.go:20 +0x7ca
	/var/lib/jenkins/go/pkg/mod/github.com/bsergean/[email protected]/torrent/torrent_run.go:20 +0x7ca
	/var/lib/jenkins/go/pkg/mod/github.com/bsergean/[email protected]/torrent/torrent_run.go:20 +0x7ca

All torrents except one are waiting on line torrent_run.go:20, which is the select statement. That means, their torrent loop is ready to receive messages from outside and process it. This is how this health check mechanism determines the torrent is health and processing messages.

However, one loop is executing torrent_run.go:86. When I look the whole traceback of that goroutine, I see that it is in runnable state and printing a debug log statement. It must have been stuck for 60 seconds while trying to print that log statement. Underneath the logging library, it calls runtime.Caller() function to find the filename to inject into the log statement.

My guess is that there is too much load on the server so this goroutine might never found the chance to run.

Another guess would be the process is suspended for a duration longer than 60 seconds and resumed later. That's why health checker mechanism decided that the torrent is not health and crashes with the stack traces.

Any thoughts?

from rain.

cenkalti commented on May 28, 2024

If it is frequently the logging case, you can try to lower your log volume by setting the level to INFO or WARNING to reduce the chance of crash as workaround. I don't recommend using DEBUG level at production as it is producing too much log and that may slow down the execution of the program.

from rain.

bsergean commented on May 28, 2024

Thanks for looking at my big tracebacks :) ! When I look at logs the level feels reasonable, my own program might log a little too much, but nothing too crazy either. The machines were my program runs can have a fairly high load, but 60 seconds sounds like a lot of time. Maybe there’s a lock somewhere deep in the logging library ? I use (and abuse) the logrus library in my program, with some locks too (so that I can capture all logs and server them in a health-check endpoint, which is very convenient for debugging/inspecting). That could cause some bad interactions.

…

On Oct 31, 2021, at 10:00 AM, Cenk Altı ***@***.***> wrote: If it is frequently the logging case, you can try to lower your log volume by setting the level to INFO or WARNING to reduce the chance of crash as workaround. I don't recommend using DEBUG level at production as it is producing too much log and that may slow down the execution of the program. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#38 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC2O6UNLW6DEHNSSOPEYNV3UJVY2BANCNFSM5GATE5XA>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

from rain.

github-actions commented on May 28, 2024

This issue is stale because it has been open for 30 days with no activity.

from rain.

github-actions commented on May 28, 2024

This issue was closed because it has been inactive for 14 days since being marked as stale.

from rain.

Health check crashes while printing a log statement about rain HOT 5 CLOSED

Comments (5)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	// checkTorrent pings the torrent run loop periodically and crashes the program if a torrent does not respond in
	// specified timeout. This is not a good behavior for a production program but it helps to find deadlocks easily,
	// at least while developing.
	func (s Session) checkTorrent(t torrent) {