adamwiggins / stalker Goto Github PK

The worker exited after this exception, why didn't the error handler catch it?

[Thu Sep 09 19:04:32 +0000 2010] -> instance.launch app/models/instance.rb:70:in `launch': Instance::RunawayLaunches (Instance::RunawayLaunches)
from ./jobs.rb:24
from .bundle/gems/stalker-0.4.2/lib/stalker.rb:62:in `call'
from .bundle/gems/stalker-0.4.2/lib/stalker.rb:62:in `work_one_job'
from .bundle/gems/stalker-0.4.2/lib/stalker.rb:53:in `work'
from .bundle/gems/stalker-0.4.2/lib/stalker.rb:53:in `loop'
from .bundle/gems/stalker-0.4.2/lib/stalker.rb:53:in `work'
from .bundle/gems/stalker-0.4.2/bin/stalk:19
from .bundle/bin/stalk:19:in `load'
from .bundle/bin/stalk:19

arion@f4fd698

HI
I'm using stalker on engineyard hosting for sending emails in a background. So generally it works fine but after some time it stops to send emails. I check logs and see errors
"Mysql server has gone away"
What I'm doing wrong?

thanks in advance

Noticed it when working with bundler:

1.8.7 stealth:vc_whitebox jcobbett$ bundle exec stalk app/workers/card_data.rb
/Users/cobbo/projects/vc_whitebox/lock/gems/stalker-0.2.4/bin/../lib/stalker.rb:2:in `require': no such file to load -- json (LoadError)

job "address.geocode" do |args|
    address = Address.find(args["id"])
    address.geocode
    address.save
end

does work. But when using address.save :validate => false the same job gets processed over and over again until it fails.

Working address.geocode (id=4)
Finished address.geocode in 233ms 
Working address.geocode (id=4)
Finished address.geocode in 321ms 
Working address.geocode (id=4)
Finished address.geocode in 221ms 
Working address.geocode (id=4)
Finished address.geocode in 188ms 
Working address.geocode (id=4)
...


Idk if it's just me or if this is a bug. However I would mention this in the wiki.

I was trying stalker and when I try to start a worker I get Beanstalk::InvalidTubeName . I am using ruby 1.8.6 rails 2.3.3 and beanstalk-client 1.1.0 and stalker 0.9.0

Here is the full trace

/home/jetset/.rvm/gems/ruby-1.8.6-p420/gems/beanstalk-client-1.1.0/lib/beanstalk-client/connection.rb:146:in watch': fetch_trends (Beanstalk::InvalidTubeName) from /home/jetset/.rvm/gems/ruby-1.8.6-p420/gems/beanstalk-client-1.1.0/lib/beanstalk-client/connection.rb:382:insend'
from /home/jetset/.rvm/gems/ruby-1.8.6-p420/gems/beanstalk-client-1.1.0/lib/beanstalk-client/connection.rb:382:in call_wrap' from /home/jetset/.rvm/gems/ruby-1.8.6-p420/gems/beanstalk-client-1.1.0/lib/beanstalk-client/connection.rb:412:insend_to_all_conns'
from /home/jetset/.rvm/gems/ruby-1.8.6-p420/gems/beanstalk-client-1.1.0/lib/beanstalk-client/connection.rb:424:in map_hash' from /home/jetset/.rvm/rubies/ruby-1.8.6-p420/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:inmap'
from /home/jetset/.rvm/gems/ruby-1.8.6-p420/gems/beanstalk-client-1.1.0/lib/beanstalk-client/connection.rb:424:in each' from /home/jetset/.rvm/gems/ruby-1.8.6-p420/gems/beanstalk-client-1.1.0/lib/beanstalk-client/connection.rb:424:inmap'
from /home/jetset/.rvm/gems/ruby-1.8.6-p420/gems/beanstalk-client-1.1.0/lib/beanstalk-client/connection.rb:424:in map_hash' ... 7 levels... from /home/jetset/.rvm/gems/ruby-1.8.6-p420/gems/stalker-0.9.0/lib/stalker.rb:63:inwork'
from /home/jetset/.rvm/gems/ruby-1.8.6-p420/gems/stalker-0.9.0/bin/stalk:21
from /home/jetset/.rvm/gems/ruby-1.8.6-p420/bin/stalk:19:in `load'
from /home/jetset/.rvm/gems/ruby-1.8.6-p420/bin/stalk:19

I'd like to see something like :

response = Stalker.enqueue_with_response("sum", :a => 1, :b => 2)
or 
Stalker.enqueue_with_response("sum", :a => 1, :b => 2) do |response|
  puts "1 +2 = #{response['sum']}"
end

and the worker:

job_with_response "sum" do |args| 
  return :sum => args['a'] + args['b']
end

This would work by adding something like :respond_to => "stalker-response-#{$$}-#{SecureRandom.uuid}" to the job params, then subscribing to that queue (perhaps with a configurable timeout) where the worker will then enqueue it's return value once finished.

There could also be partial responses so one could do something like:

job_with_response "sum" do |args, response|
  response['status'] = 'going to calculate!'
  response['status_pct'] = 0
  response['sum'] = args['a'] + args['b']
  response['status'] = 'done calculating!'
  response['status_pct'] = 100
end

Getting a little carried away here.

Hi,

This isn't exactly an issue but i am wondering if there is a way to add a job to the stack but set the timeout manually. I have everything configured to process jobs for video transcoding however these jobs take a long time so I need to set the Timeout to a few hours or so.

Thank you advance.

Currently, stalker manage only 1 beanstalkd server. It is hard coded to initialize a pool of 1 server. (Beanstalk::Pool.new([ beanstalk_host_and_port ])).

Could you add the possibility to set a collection of server based on a configuration?

Thanks,