• http://github.com/mjrusso/pyres/commit/341046c7923f4f2e6a6500ee07ada583e5a9aef3
• http://github.com/mjrusso/pyres/commit/1cbe8a537e2b9b4753e573ef45710a365aeabde4

i'd like to add support for running resweb through other wsgi containers like gunicorn, gevent, tornado, and others.

instead of polling, use the bpop methods as part of redis 2.0

We're using the failed items queue quite a lot at the moment.

There are several things that would be useful:

• Ability to reference failed jobs by a url/id
• Javascript requeue/delete
• Multi select and then requeue/delete

It'd also be good to remember the page you were on when you hit requeue/delete but that's probably because our error list is massive right now :)

Not sure what the consensus on using javascript is? Might be able to help out with a patch at some point.

there are some newer functions that have no tests written yet

stacktraces for failures are not displaying all the information. Need to fix

A number of features of the web interface were broken. A series of patches that correct these issues:

• http://github.com/mjrusso/pyres/commit/9e542825c67b65e4361a40b51e1c17c686ab7495
• http://github.com/mjrusso/pyres/commit/ea55269163bda640e74ebb16bc8488958b972882
• http://github.com/mjrusso/pyres/commit/d7a43ae63e6c7fc6dcbdbcb20cd34b8c1df54aec
• http://github.com/mjrusso/pyres/commit/fc07fb89c0ed3cd427708f51966e3b413268071f
• http://github.com/mjrusso/pyres/commit/f949b01512f8095e2299bb993e81e03e0003edd1
• http://github.com/mjrusso/pyres/commit/8423b23ada07a0b4b06cc13b0cee7ecf1d6c6508

The delete all view directly removes the failed queue from Redis. This means that the stats don't get updated.

I noticed that there is a package called setproctitle when gunicorn started using it recently. (See gunicorn commit: http://github.com/benoitc/gunicorn/commit/5afec8d7268b081ba2f1a806b08bd2418519aa59 and package details: http://pypi.python.org/pypi/setproctitle/ )

It would be very nice if the name of pyres workers (parents and children) took advantage of this to convey information. Examples from Resque:

$0 = "resque: Starting"

$0 = "resque: Forked #{@child} at #{Time.now.to_i}"

$0 = "resque: Processing #{job.queue} since #{Time.now.to_i}"

$0 = "resque: Paused"

$0 = "resque: Waiting for #{@queues.join(',')}"

need to create a new package with all the fixes/upgrades that have come in since 0.5. I'm going to update the package to version 0.7 because of the extra updates.

I'm working on adding some failure backend support past the current redis-flavored failure handling. My first goal is hoptoad, but I might also build an email backend and some other stuff. Additionally, I'm going to offer the backend agnostic API that Resque currently features.

In line with project goals, I'm going for verisimilitude with Resque while striving to be pythonic, whatever that means.

I'd like to remove most of the code in pyres from the init into modules where it makes sense. This is a style and maintenance thing for me. I also think this might help with the jobhook and extension issues.

a grey background is all that shows up for resweb when using chrome.

I have a python project where I have {pyres, resweb, scripts} in resque folder. Whenever I put a job in the queue I do

>>> from resque.pyres import ResQ
>>> r = ResQ()
>>> r.enqueue(Someclass, "hello")

I am sure "Someclass" class has a perform method on it which prints the arg. But when the job is picked up it fails with error

NoneType has no perform method

When I have pyres in my project root it works. Just dont get why this is failing.

(Thanks for fixing a previous logging bug with http://github.com/binarydud/pyres/commit/a5a0b603d059e6c11f3fd24c9a88e7001ca3919e)

There's another issue related to the logging of job arguments.

You would get this when trying to enqueue a job with more than one arg:

Traceback (most recent call last):
   ...
  File "/Users/mjr/Desktop/pyres/pyres/pyres/__init__.py", line 126, in enqueue
    logging.debug("job arguments: %s" % args)
TypeError: not all arguments converted during string formatting

(This happens because args is a tuple with length > 1.)

I modified a test to cause the current code to break, and checked in the fix to my fork here:

http://github.com/mjrusso/pyres/commit/e9440805f0268b73cecc377ad66dd20c6e1883e7

I need to get off my lazy ass and write some actual sphinx docs for this project. However, right now i will settle for a slightly better README.

there is a newer version of the redis-py server that has some support for upcoming features like blocking pop.

Is there a way to start multiple workers.

$ ./pyres_worker queue1, queue2

Seems to start a worker which checks queue1, queue2 and sleep for 5. In production we need to have ability to start multiple workers / queue basis and keep the process under supervisord or some process monitor

the resweb django project needs to be finished

In the same vein as requeuing items from the failed queue, we need to be able to delete items from the queue.

see #15

Instead of just having stats on the workers, we should have stats like processed that exist per queue.

I'd like to be able to easily access the redis instance that a job came from, within the perform method of the job class.

recent builds put up on pypi seem to be lacking the package_data fro resweb. For now i've pointed the download to the github link for each version tag.

I didn't see this one before because the behavior changes depending on the pid(s) of the process(es).

• http://github.com/mjrusso/pyres/commit/6390ac7ae199d49a7bba593b24c2c45a83c3f400
• http://github.com/mjrusso/pyres/commit/dfdbe2ff217d915cdf78c2ba5f007408e4ce73bf

ResQ object needs to have the keys method implemented

It'd be nice to be able to delete all the items from the error queue rather than going through and deleting them individually (similarly it'd be good to be able to requeue all).

Not sure if this is in the pipeline or easy. I'm using pyres 0.8

merge the following commits into master:

• http://github.com/mjrusso/pyres/commit/7c80dfa450c5834d455eaa417dde9f1bf5553508
• http://github.com/mjrusso/pyres/commit/6e937a28d9dd5231e42bd229b90cb17952fa3478

I was lazy and put print statement throughout the code. I need to modify the code to use the built-in logging module.

I saw this in the README: "I'd like to get this as close to the resque web interface as possible."

I've been meaning to port the Resque web interface over to use Mustache templates. If that happens, you could use Pystache and we could share both html and css.

I'd also volunteer to help work on the web interface, as I've been looking for excuses to do some on-the-side Python web hacking.

What do you think? Any interest in sharing templates and whatnot?

Sometimes a failed item needs to be re-queued because the issue was a path problem with the worker.

Figured it might be a good idea to add some guidelines about contributing to pyres.

the scheduled tasks branch is almost ready for master. I am going to try and add some more tests about the generators and the pop methods before I do the merge.

when a delayed queue item is put back on the main redis queue, it should be pushed at the front of the list instead of the back of the list.

Or something like that.

http://github.com/mjrusso/pyres/commit/e15a3e7f3dc095b2ed318df33c13dded500903d8

A straight-up port of Resque's prune_dead_workers and worker_pids methods.

Now, if you kill -9 your workers on a given machine (or something much, much worse happens), the next worker to start up on that same machine will unregister the dead ones.

Inlcude better options than just logging to the console. Should have rotatable logs or logs that self rotate after a certain size.

Itty is nice for small webapps, but there are things that I'd like to do on resweb, such as flash messages. Flask is the first idea that popped into my head, but I would be willing to look at other options.

pyres 1.0 will be 2.0 compatible. This is primarily a redis-py upgrade.

There needs to be a way to include before and after job hooks and perhaps after start and before end and such. This would allow for stuff like persistent db/socket/mail connections from workers.

ResQ needs a implementation of the "working" method

certain views need pagination. Examples include the failures list and the queue listings

When I switch to the v0.5.0 tag and install the pyres package via setup.py, pyres_web works fine.

When I switch to the current master and and install the pyres package via setup.py, pyres_web crashes when attempting to serve a page. Take a look:

$> pyres_web
itty starting up (using wsgiref)...
Listening on http://localhost:8080...
Use Ctrl-C to quit.

<type 'exceptions.IOError'> occurred on '/': [Errno 2] No such file or directory: '/Library/Python/2.6/site-packages/pyres-0.5.0-py2.6.egg/resweb/templates/overview.mustache'
Traceback: Traceback (most recent call last):
  File "/Library/Python/2.6/site-packages/itty-0.6.4-py2.6.egg/itty.py", line 225, in handle_request
    response = callback(request, **kwargs)
  File "/Library/Python/2.6/site-packages/itty-0.6.4-py2.6.egg/itty.py", line 344, in new
    return method(*args, **kwargs)
  File "/Library/Python/2.6/site-packages/pyres-0.5.0-py2.6.egg/resweb/server.py", line 23, in index
    return str(Overview(HOST).render())
  File "build/bdist.macosx-10.6-universal/egg/pystache/view.py", line 95, in render
    template = self.load_template()
  File "build/bdist.macosx-10.6-universal/egg/pystache/view.py", line 61, in load_template
    f = open(self.template_file, 'r')
IOError: [Errno 2] No such file or directory: '/Library/Python/2.6/site-packages/pyres-0.5.0-py2.6.egg/resweb/templates/overview.mustache'
localhost - - [22/Feb/2010 00:18:09] "GET / HTTP/1.1" 500 17

This might be related to issue #12.

I added the following line to setup.py and the interface now works fine:

package_data={'resweb': ['templates/*.mustache','media/*']},

This is a low-priority feature request, but it would be convenient to augment the Queue view to show the last x successfully completed jobs and last x failed jobs (in addition to just the pending jobs, as we have today).

The Job class perform method should throw an exception if it can't load the worker class. At the moment safe_str_to_class returns NoneType which then causes an error which doesn't make it immediately obvious what the problem is.

merge update from aezell into master

I think that pyres should be compatible with resque.

I should be able to use pyres_worker to create Python workers that can subsequently be monitored via the resque web interface. Conversely, I should be able to use rake resque:work to create a Ruby worker that can then be monitored by the pyres web interface.

This is important because there are a number of alternative worker implementations being built against resque -- for example, in the C, C#, Scala languages.

Right now, I believe that the only issue barring compatibility is the timestamp format. The pyres web interface crashes if a job is created by a Ruby resque worker, because it reads a timestamp in a format it is not expecting. (The resque web interface reports all dates/times as NaN for all jobs created by Python pyres workers.)

This could trivially be fixed by changing the pyres timestamp format to match resque's.

I can make the change in my fork, but I would like to know if this will make it into the mainline.

There's a typo in this file: http://github.com/binarydud/pyres/blob/master/resweb/templates/header.mustache

On line 7 the script tag isn't closed properly.

Only just picked this up because I started using Chrome 7 (and it just displays a grey screen for any pyres page). Has been fine in other versions of chrome and in firefox.

Cheers

Rob

This is looking good! What's the reason for using klass in the payload instead of class?

Just asking because using class would make pyres compatible with Resque and the Scala version of Resque.

it would be nice to be able to use not only args, but also kwargs in the enqueue method. As far as I can tell, the only problem that might pop up would be enqueuing something with kwargs, but the job's perform method doesn't actually take any keyword arguments. Of course, this same issue applies to positional arguments as well.

move to itty and jinja2 for the resweb module

It seems that delayed_queue_schedule_size() relies on the cardinality of the resque:delayed_queue_schedule ZSET. This means that any jobs that occur within the same second/timestamp will only be counted as a single job, thereby producing an incorrect return value.