Workers do not respond appropriately to SIGTERM during activerecord mysql queries about backburner HOT 10 CLOSED

Interesting, I can see why this would be a problem. Do you have thoughts on the best way for me to handle this case? How do I know a ActiveRecord::StatementInvalid should be read as a SIGTERM?

from backburner.

Wow, thanks for the quick response.

I think if you trap the TERM, it won't be caught by rails and turned into StatementInvalid. So, borrowing from the stackoverflow thread I linked, if this were somewhere in the worker:

Signal.trap("TERM") do
Kernel.exit!
end

That would make it exit immediately. I thought about just throwing this into the worker and submitting a pull request, but the problem with this approach is that it terminates the worker in the middle of the job. I imagine the ideal behavior would be for the worker to only exit after the current job is finished processing, yes?

from backburner.

It actually doesn't really matter, if the job doesnt finish processing, it will eventually get auto released and then reprocessed by another worker in the future. So I think that's a non issue right?

from backburner.

Ah, true. I'm not yet accustomed to relying on the magic of beanstalkd :)

from backburner.

Yeah beanstalkd makes it really hard to lose jobs, you have to go out of your way :)

from backburner.

Can you try this from master and verify it works?

from backburner.

The above being said, I would want to have it finish processing the job eventually since if it's literally halfway through a complex job, it might redo some work twice if it doesn't get to finish. But I might tackle that as a separate issue.

from backburner.

Wow, that was fast :) we're using master now. this bug was very
intermittent so it's a little bit hard to say with certainty that the issue
is fixed, but it looks good!

Tyler Kiley
InQuicker, LLC
[email protected] | 770-597-9185

On Mon, Apr 22, 2013 at 10:03 PM, Nathan Esquenazi <[email protected]

wrote:

Can you try this from master and verify it works?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/22#issuecomment-16836640
.

from backburner.

Ok released 0.3.4 with this fix included. Thanks, hopefully that problem goes away.

from backburner.

@nesquena is there a means for gracefully shutting down (give worker's current job some time to finish and kill worker after it does)

Retrying a job that times out due to a worker being killed is not a big deal with idempotent jobs (or jobs whose side effects are guarded within a transaction), but for non-idempotent jobs (like sending an email or push notification) it'd be really nice to know there's little chance of a redeploy killing a worker in the time between when an e-mail was sent and when backburner notifies beanstalkd that the job was released/completed. Thanks for any insight and thanks for this awesome library (and rabl!)!

from backburner.