Giter VIP home page Giter VIP logo

resque-job-stats's Introduction

resque-job-stats

Build Status

Job centric stats for Resque.

Stats are tracked per Job type (class/module) in addition to the worker based stats Resque provides.

Stats tracked are:

  • Jobs performed
  • Jobs enqueued
  • Jobs failed
  • Duration of last x jobs completed
  • Average job duration over last 100 jobs completed
  • Longest job duration over last 100 jobs completed
  • Jobs enqueued as timeseries data (minute, hourly)
  • Jobs performed as timeseries data (minute, hourly)

This information can be used to help track performance and diagnose specific bottlenecks.

We are sending this information to Nagios for graphing and alerts (via a custom rake task).

Installation

Requires resque '>= 1.17', '< 3'

In your Gemfile add:

gem 'resque-job-stats'

Usage

Simply extend your class

class MyJob
  extend Resque::Plugins::JobStats

  @queue = :my_job
  def self.perform(*args)
    # ..
  end
end

And you will have a set of keys starting with 'stats:jobs:my_job' inside your Resque redis namespace.

Alternatively you can include just the metric you wish to record.

class MyVerboseJob
  extend Resque::Plugins::JobStats::Performed
  extend Resque::Plugins::JobStats::Enqueued
  extend Resque::Plugins::JobStats::Failed
  extend Resque::Plugins::JobStats::Duration
  extend Resque::Plugins::JobStats::Timeseries::Enqueued
  extend Resque::Plugins::JobStats::Timeseries::Performed

  @queue = :my_job
  def self.perform(*args)
    # ...
  end
end

Duration module

The duration module provides two metrics, the longest job and the job rolling avg.

These are accessible via two singleton methods, MyJob.job_rolling_avg and MyJob.longest_job.

By default the last 100 jobs durations are stored and used to provide the above metrics.

You may set the number of jobs to include by setting the @durations_recorded variable.

class MyJob
  extend Resque::Plugins::JobStats::Duration

  @queue = :my_job
  @durations_recorded = 1000

  def self.perform(*payload)
    # ...
  end
end

Timeseries module

The timeseries module provides timeseries counts of jobs performed. The metrics are rolling and kept for a period of time before being expired. The timestamp used for the timeseries data is UTC.

Resque Web Tab

The Resque web page for showing the statistics will only display jobs that extend Resque::Plugins::JobStats (in other words, just the jobs that include all of the metrics):

class MyJob
  extend Resque::Plugins::JobStats
  ...
end

The interface can be included in your app like this:

require 'resque-job-stats/server'

If you wish to display only certain metrics, you can filter the metrics accordingly. The default metrics can be found in Resque::Plugins::JobStats::Statistic.

  Resque::Server.job_stats_to_display = [:jobs_enqueued, :job_rolling_avg]

Screenshots

Overview

overview stats

Individual Job Histories

individual stats

TODO

  • Find clean way to add queue wait time stats.

Contributing to resque-job-stats

  • Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet
  • Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it
  • Fork the project
  • Start a feature/bugfix branch
  • Commit and push until you are happy with your contribution
  • Make sure to add tests for it. This is important so I don't break it in a future version unintentionally.
  • Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.

Contributers

Copyright

Copyright (c) 2011-2012 Alan Peabody. See LICENSE.txt for further details.

resque-job-stats's People

Contributors

alanpeabody avatar damonmorgan avatar djburdick avatar djmaze avatar jesperronn avatar lukeasrodgers avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

resque-job-stats's Issues

UI to display the timeseries data

This data doesn't seem to be included in the UI:
Jobs enqueued as timeseries data (minute, hourly)
Jobs performed as timeseries data (minute, hourly)

Add to the Job_Stats UI view

release new gem?

Fantastic little gem, very useful. I had installed this via rubygems and got a version that, when extending my classes with JobStats, didn't provide all the stats as listed in the README. But grabbing directly from the master branch of the repo brought in those missing parts.

May be time to freshen the version up at rubygems.org?

Extending a superclass means subclasses won't get job stats

I've done this as a workaround in my BaseJob superclass:

# JobStats is implemented in such a way that we can't just extend a superclass:
# https://github.com/alanpeabody/resque-job-stats/blob/master/lib/resque/plugins/job_stats.rb#L19-L21
def self.inherited(subclass)
  subclass.extend Resque::Plugins::JobStats
end

Not expecting this to be fixed since the lib appears not to be maintained anymore, but perhaps this helps someone else.

Durations key screws the failed key

i'd do a pull request but i've already screwed compatibility and this fix is pretty trivial
doh:
def jobs_duration_key
@jobs_failed_key ||= "stats:jobs:#{self.name}:duration"
end

Maintainer Needed.

I am no longer using Resque or this library. I am happy to hand of maintenance to anyone who is actively using this.

Jobs run twice to measure duration for duration and history modules

In modules Resque::Plugins::JobStats::Duration and Resque::Plugins::JobStats::History I see two different callbacks that start from around_perform_? From what I understand it means that job run twice to measure the time, is it intentional behaviour?

I'm trying to make this gem work with ActiveJob and this is how I can make around_perform_ callbacks work

         base.around_perform do |job, block|
            job.class.methods.select do |meth|
              meth.to_s.start_with?("around_perform_")
            end.each do |meth|
              job.class.send(meth) { block.call }
            end
          end

but in this case if we have more than 1 callback we call job few times. Could we discuss it?

Warning from Resque about Resque::Helpers

Hey there,

Will this be addressed soon?
I see the warning and iam using resque 1.25.2 currently.
Resque::Helpers will be gone with no replacement in Resque 2.0.0

Thanks
Jag

Help testing against edge resque

Hey there!

I'm gearing up to work on Resque 2.0, and I'd like to coordinate better with plugin authors to make sure stuff doesn't break.

I'd like to know a few things:

  1. Can I do something to help get you testing against edge Resque?
  2. Are you monkey-patching anything in Resque currently? Can 2.0 expose an API to help you not have to do that any more?
  3. Do you need any help in bringing your plugin up-to-date with the latest Resque?

Thanks!

Related: https://github.com/defunkt/resque/issues/880

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.