Hey there!

I'm gearing up to work on Resque 2.0, and I'd like to coordinate better with plugin authors to make sure stuff doesn't break.

I'd like to know a few things:

1. Can I do something to help get you testing against edge Resque?
2. Are you monkey-patching anything in Resque currently? Can 2.0 expose an API to help you not have to do that any more?
3. Do you need any help in bringing your plugin up-to-date with the latest Resque?

Thanks!

Related: https://github.com/defunkt/resque/issues/880

Using Redis Sets for batch meta data was a bad move on my part. Sets do not allow duplicate members. This causes problems when batching jobs with identical arguments.

When modifying batch data, we need to implement key locking to prevent race conditions.

Manage batch statistics like success/failure counts and any other meta data.

I was just wondering if there were any plans to support the latest Resque version. I'm using 2.0.0.pre.1 and I ran into an error that I fixed with a small one line fix. I know Resque is in a kind of weird limbo between supporting 1.x and the new 2.0 version.

In our environment we have two workers that process jobs. When all the jobs in a batch complete we want to fire off a number of additional tasks. Unfortunately we sometimes see the after_batch_hook called multiple times.

Issue: multiple workers
For example here are two nearly contiguous lines from our log file:

after_batch_hook, job_code: retry_32110_20130211T003058.591Z_19236, science-example-com, 2013-02-07, {}
...
after_batch_hook, job_code: retry_32110_20130211T003058.591Z_19236, scrapbooking-example-com, 2013-02-07, {}

The occur within minutes of each other and are clearly being fired off by each of the workers. I think there may be a race condition when multiple workers grab jobs because the queue could be empty at the time that they each check.

Issue: multiple calls to the hook
Additionally, we sometimes see the hook called hourly. It's very strange that nearly every hour it calls this hook again. For example, an hour later in the same logs we see this call:

after_batch_hook, job_code: retry_32110_20130211T003058.591Z_19236, www-example-com, 2013-02-07, {}

Note that the batch ID is exactly the same as the previous message. I don't even know where to start looking at the cause of this second issue.

Thoughts?

Hey there!

I'm gearing up to work on Resque 2.0, and I'd like to coordinate better with plugin authors to make sure stuff doesn't break.

I'd like to know a few things:

1. Can I do something to help get you testing against edge Resque?
2. Are you monkey-patching anything in Resque currently? Can 2.0 expose an API to help you not have to do that any more?
3. Do you need any help in bringing your plugin up-to-date with the latest Resque?

Thanks!

Related: https://github.com/defunkt/resque/issues/880

I'm wondering what you think of the following potential feature. I need to implement it for my requirements, so it's going to happen either way. Just trying to figure out if I should add it to the gem or my own, custom extension in our codebase.

Basically, the idea would be to specify a certain number of failed jobs that a batch would accept and still execute the after_batch hooks. For instance, say you had 20 jobs in your batch named test_batch. You want after_batch_send_email to execute if 18 or more jobs complete.

My current thoughts are the following: add an on_failure hook that creates/adds a value to a redis list key named something like batch:batch_id:failures that is essentially like the current batch:batch_id list, except it obviously tracks failures instead of what is left to be processed. Also on_failure and after adding its value to the failed batch list, it checks if a) the failure list is shorter than a given variable (say @acceptable_failure_count) and b) the values in the batch:batch_id:failures list are the exact same values as the ones in batch:batch_id. If so, that means each batch that is left in batch:batch_id has failed and no other batch jobs are left to be processed. If both conditions are true, we trigger the after_batch_hooks and remove the batch keys, otherwise we ignore the hooks.

Ability to limit one worker for a batch? Obviously I haven't given this much thought.

Note: This might go hand in hand with issue #14.

Write a failure backend that handles batched job duties.

I've created for myself a module called Resque::Plugins::BatchedJobRetry which fixes an incompatibility between resque-batched-job and resque-retry and was wondering if that's something people might like integrated into this gem. The two gems will fail spectacularly together if any job in the batch retries even once, as it will add another job to the batch for the job that is requeued to retry, but on completion it can only remove one job from the batch. Thus, the batch is stuck forever. Here's the actual code to fix this, it's quite simple.

module Resque
  module Plugins
    module BatchedJobRetry
      include Resque::Plugins::BatchedJob
      include Resque::Plugins::Retry

      # Resque hook that handles batching the job. (closes #2)
      #
      # @param [Object, #to_s] id Batch identifier. Any Object that responds to #to_s
      def after_enqueue_batch(id, *args)
        unless redis.exists redis_retry_key(id, *args)
          mutex(id) do |bid|
            redis.rpush(bid, encode(:class => self.name, :args => args))
          end
        end
      end
    end
  end
end

The one issue I see with this code is that if a new job is being added to a batch after an existing job with the same parameters has been requeued to retry it, then it won't push another job onto the batch list and thus only one of them might finish before the batch is complete. This seems to be an issue with the retry gem in general, however, as any two jobs with the exact same parameters will share a retry key in redis unless the identifier function is overridden to provide a unique identifier for each job (which essentially boils down to having unique parameters anyway).

Quite often I'll find one or two jobs left in the batch which have completed successfully. I have to recreate them so they will finish and fire the after_batch callback.

Is there any reason this could happen? How can I debug further? Thanks!

How does one use resque-batched-job with activejob? My self.after_batch_finished hook never gets invoked.

require 'resque/batched_job'

class MyJob < ActiveJob::Base
  extend Resque::Plugins::BatchedJob
  def perform(bid, args)
    # Perform job.
  end
  def self.after_batch_finished(bid, *args)
    # Never reaching this.
  end
end

What am I doing wrong?

From time to time we'll have jobs that all complete, but that after_batch callback won't fire. When we look further, we'll see that the batch queue in redis still has entries (which is what kept the callback from firing) and we're not sure why this is happening. None of the jobs failed (even though failed jobs are supposed to remove themselves from the batch queue) and there is nothing else we can see that is causing this to happen.

Any ideas?

Our stack:

Rails 3.0.10
Resque 1.19.0
Resque Batched Job 0.1.5

Let me know if you need more information. Thanks!

Requiring resque-batched-job generates deprecation warnings with current resque 1.x versions, due to the inclusion of Resque::Helpers:

$ egrep resque Gemfile
gem 'resque', '1.25.2'
gem 'resque-batched-job', '1.7.2'

$ irb
2.1.2 :001 > require 'resque'
 => true
2.1.2 :002 > require 'resque-batched-job'
Resque::Helpers will be gone with no replacement in Resque 2.0.0.
 => true

Implement a method that removes a job from a batch list.

Provide support for batched job management like failures.