michaelcameron / grails-jesque Goto Github PK

View Code? Open in Web Editor NEW

26.0 26.0 19.0 763 KB

Grails Jesque Plugin

Groovy 57.45% CSS 7.18% JavaScript 0.09% Java 32.46% Batchfile 2.82%

grails-jesque's People

Contributors

Stargazers

Watchers

Forkers

tednaleid rafanami colinharrington spantree jonspalmer caredotcom xnickmx dnoseda ralcaraz cdeszaq peh augustine-tran jrherrera szels ctoestreich

grails-jesque's Issues

Configurable fail queue

How can i configure to have different fail queues instead of only one common?

I want a dedicated fail queue per each queue to hold their failed jobs.

Upgraded redis plugin dependency

I know that there's some awareness of this already due to the revert made in this commit:

eddd437

But I just wanted to make sure the issue had the proper visibility. I also wanted to document what I've found so far in case I'm able to create a patch to allow the upgrade.

The latest version of the Redis plugin is 1.3.1, however grails-jesque requires 1.0.0M9 for compatibility reasons. We had been using a newer version ourselves alongside jesque, but recently things started breaking. I think it's related to a newer version of the jesque client.

using redis 1.3.0 or redis 1.3.1 seems to create problems with the doEnqueue method of the jesque client:

    public static void doEnqueue(final Jedis jedis, final String namespace, 
            final String queue, final String jobJson)
    {
        jedis.sadd(JesqueUtils.createKey(namespace, QUEUES), queue);
        jedis.rpush(JesqueUtils.createKey(namespace, QUEUE, queue), jobJson);
    }

Here's the exception that's being thrown when using 1.3.0 or greater when trying to en queue a new job:

http://imageshack.us/f/151/screenshot20120523at606.png/

I'm able to adjust my BuildConfig.groovy to use redis-1.2.1 to solve the issue:

    plugins {

      ...

        compile(":redis:1.2")        // Using a newer version seems to break jesque  :-/
        compile(":jesque:0.4.0-SNAPSHOT") {
            excludes "redis"
        }
        runtime (":jesque-web:0.3.2") {
           excludes "redis","hibernate"
        }

     ...
    }

Note that we're using a newer version of grails-redis because we need the functionality of that plugin independently of the jesque plugin. Otherwise, I would stick to whatever version grails-jesque asked for.

This is on grails 2.0.3

Ability to use a class with the JesqueService.enqueue

As discussed at GUM (http://groovy.mn) The JesqueService should take a class

Error with latest version of joda-time and delayed jobs

Just installing the plugin in BuildConfig.groovy, when the project has a dependency on a newer version of joda-time, causes stacktraces on startup. I had a dependency on joda-time v2.4. It was working with joda-time v2.1

Here is the stacktrace:

15:06:04.121 ERROR {} <> [ ] (Jesque Delayed Job Thread) jesque.JesqueDelayedJobThreadService - Jesque delayed job exception, attempt 119 of 120
org.joda.time.IllegalFieldValueException: Value 292278994 for year must be in the range [-292275054,292278993]
    at org.joda.time.field.FieldUtils.verifyValueBounds(FieldUtils.java:234)
    at org.joda.time.chrono.BasicYearDateTimeField.set(BasicYearDateTimeField.java:83)
    at org.joda.time.chrono.ZonedChronology$ZonedDateTimeField.set(ZonedChronology.java:482)
    at org.joda.time.base.BaseDateTime.<init>(BaseDateTime.java:129)
    at org.joda.time.base.BaseDateTime.<init>(BaseDateTime.java:97)
    at org.joda.time.DateTime.<init>(DateTime.java:209)
    at grails.plugin.jesque.JesqueDelayedJobService$_nextFireTime_closure3.doCall(JesqueDelayedJobService.groovy:58)
    at grails.plugin.redis.RedisService.withRedis(RedisService.groovy:82)
    at grails.plugin.jesque.JesqueDelayedJobService.nextFireTime(JesqueDelayedJobService.groovy:52)
    at grails.plugin.jesque.JesqueDelayedJobThreadService.mainThreadLoop(JesqueDelayedJobThreadService.groovy:50)
    at grails.plugin.jesque.JesqueDelayedJobThreadService$_run_closure1.doCall(JesqueDelayedJobThreadService.groovy:40)
    at grails.plugin.jesque.JesqueDelayedJobThreadService.withRetryUsingBackoff(JesqueDelayedJobThreadService.groovy:75)
    at grails.plugin.jesque.JesqueDelayedJobThreadService.run(JesqueDelayedJobThreadService.groovy:39)
    at java.lang.Thread.run(Thread.java:722)

This seems to be caused by using the Long.MAX_VALUE as a date, which appears to be invalid with later versions of joda-time. Specifically in JesqueDelayedJobService.nextFireTime(). When there are no delayed jobs, the MAX_VALUE is used, which is invalid.

Support configurable cronExpressions

It's restricting that that cronExpression has to be hardcoded as a string in the trigger on the Job class.

We find that we would like to define the cronExpression somewhere like Config.groovy so that we can vary the cron schedule base on environment or other configuration state.

Add ability to enqueue_in and enqueue_at a certain time duration

It would be great to round out the feature set of resque-scheduler: https://github.com/bvandenbos/resque-scheduler
We already have support for Cron syntax but we miss the ability to enqueue_in or enqueue_at. These would be amazing features to allow items in the queue to declare retry semantics like resque-retry: https://github.com/lantins/resque-retry

Transactional Services

I just noticed that some services are marked as transactional. In the end, this is not needed, as none of the services is doing any database related work.

I'll provide a patch which fixes this.

Change Job artefact name to avoid classes with other plugins (Quartz)

The newest versions of the Quartz plugin also use the "Job" indentifier to reference their artefacts. I've listed a JIRA against them as well.
The JobArtefactHandler should register the Jesque Jobs under a different namespace (i.e. jesqueJob instead of job) so as to avoid name clashes when trying to reference the artefacts in the plugin descriptor file.

For example:
application.jobClasses.each {...}

should instead be
application.jesqueJobClasses.each {...}

Hibernate 4 support

WorkerHibernateListener is written specifically for Hibernate 3. At the very least the SessionFactoryUtils and SessionHolder classes seem to be incompatible.

It would be great to get an update that supports the new version now that it is used by Grails by default. Thanks!

Adding time zone offset caused overflow

I'm not sure why but I keep getting the following exception in my Grails app:

| Error 2014-08-05 07:05:58,338 [Jesque Delayed Job Thread] ERROR jesque.JesqueDelayedJobThreadService  - Jesque delayed job exception, attempt 17 of 120
Message: Adding time zone offset caused overflow
   Line | Method
->> 965 | convertUTCToLocal     in org.joda.time.DateTimeZone
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
|   422 | get                   in org.joda.time.chrono.ZonedChronology$ZonedDateTimeField
|   129 | <init> . . . . . . .  in org.joda.time.base.BaseDateTime
|    97 | <init>                in     ''
|   209 | <init> . . . . . . .  in org.joda.time.DateTime
|    58 | doCall                in grails.plugin.jesque.JesqueDelayedJobService$_nextFireTime_closure3
|    98 | withRedis . . . . . . in grails.plugin.redis.RedisService
|    52 | nextFireTime          in grails.plugin.jesque.JesqueDelayedJobService
|    50 | mainThreadLoop . . .  in grails.plugin.jesque.JesqueDelayedJobThreadService
|    40 | doCall                in grails.plugin.jesque.JesqueDelayedJobThreadService$_run_closure1
|    75 | withRetryUsingBackoff in grails.plugin.jesque.JesqueDelayedJobThreadService
|    39 | run                   in     ''
^   744 | run . . . . . . . . . in java.lang.Thread

I've run FLUSHDB on the redis cli, there actually shouldn't be any jobs trying to run, so I'm a little confused as to what's causing this issue. It started at some point after it had been working fine for a long time. So needless to say, I'm stumped atm. The exception happens in the Jesque code, so it's not something I'm interfering with as such, and I don't specifically fiddle with TZ's or anything in the Jesque code.
Any suggestions? :-(

Failed jobs on Tomcat stop

Hi.
We are using:

grails 2.4.4
grails-jesque 0.8.0-SNAPSHOT

After stopping our tomcat server while there are unprocessed elements in the queue (around 8000), we got a lot of them (2879) in the failed queue. The error is same for all of them:

Method on class [com.sample.Subscriber] was used outside of a Grails application. If running in the context of a test using the mocking API or bootstrap Grails correctly.
    at com.sample.Subscriber.currentGormInstanceApi(Subscriber.groovy)
    at com.sample.Subscriber.save(Subscriber.groovy)
    at com.sample.Subscriber$save.call(Unknown Source)
    at com.sample.SubscriberService.saveSubscriber(SubscriberService.groovy:27)
    at com.sample.SubscriberService$$FastClassBySpringCGLIB$$64298a18.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:708)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)

....

Is there any way to avoid this issue ?

Thanks.

Can the jobs be marked as @Transactional

Hi,

This plugin is great and works nicely.

Could i just mark the job as transactional and expect it manage the perform payload code in a separated transaction with rollback on failure? ... i've trying to find documentation about this but didn't see it.

Thanks in advance.

Exceptions thrown in jobs result in Cast Exceptions in Jesque 0.3.0

We recently ran a process that threw very large numbers of jobs onto various queues. For some reason a Redis Socket Timeout error was thrown. After that a large number of jobs started failing with cast exceptions like this:

java.lang.ClassCastException: java.util.ArrayList cannot be cast to java.lang.Long

011at redis.clients.jedis.Connection.getIntegerReply(Connection.java:178)

011at redis.clients.jedis.Jedis.sadd(Jedis.java:1161)

It looks almost exactly like the case described here: redis/jedis#176

Can this situation be handled gracefully by Jesque?

Is version 0.4.0 likely to be any more robust in this regard?

Jesque classes not reloadable

This is related to GRAILS-7799, but I figured it seemed fitting to report here as well incase there's a change that needs to be made in the plugin.

When running the project on grails 2.1.0, grails-jesque 0.4.0, I get the following warning.

GRAILS-7799: Subtype 'grails.plugin.jesque.JesqueConfigurationService$$EnhancerByCGLIB$$f254165e' of reloadable type grails.plugin.jesque.JesqueConfigurationService is not reloadable: may not see changes reloaded in this hierarchy (please comment on that jira)
GRAILS-7799: Subtype 'grails.plugin.jesque.JesqueConfigurationService$$EnhancerByCGLIB$$f254165e' of reloadable type grails.plugin.jesque.JesqueConfigurationService is not reloadable: may not see changes reloaded in this hierarchy (please comment on that jira)
GRAILS-7799: Subtype 'grails.plugin.jesque.JesqueConfigurationService$$EnhancerByCGLIB$$f254165e' of reloadable type grails.plugin.jesque.JesqueConfigurationService is not reloadable: may not see changes reloaded in this hierarchy (please comment on that jira)
GRAILS-7799: Subtype 'grails.plugin.jesque.ScheduledJobDaoService$$EnhancerByCGLIB$$8f45c9d7' of reloadable type grails.plugin.jesque.ScheduledJobDaoService is not reloadable: may not see changes reloaded in this hierarchy (please comment on that jira)
GRAILS-7799: Subtype 'grails.plugin.jesque.TriggerDaoService$$EnhancerByCGLIB$$61a6907b' of reloadable type grails.plugin.jesque.TriggerDaoService is not reloadable: may not see changes reloaded in this hierarchy (please comment on that jira)
GRAILS-7799: Subtype 'grails.plugin.jesque.JesqueConfigurationService$$EnhancerByCGLIB$$f254165e' of reloadable type grails.plugin.jesque.JesqueConfigurationService is not reloadable: may not see changes reloaded in this hierarchy (please comment on that jira)
GRAILS-7799: Subtype 'grails.plugin.jesque.ScheduledJobDaoService$$EnhancerByCGLIB$$8f45c9d7' of reloadable type grails.plugin.jesque.ScheduledJobDaoService is not reloadable: may not see changes reloaded in this hierarchy (please comment on that jira)
GRAILS-7799: Subtype 'grails.plugin.jesque.TriggerDaoService$$EnhancerByCGLIB$$61a6907b' of reloadable type grails.plugin.jesque.TriggerDaoService is not reloadable: may not see changes reloaded in this hierarchy (please comment on that jira)
GRAILS-7799: Subtype 'grails.plugin.jesque.JesqueConfigurationService$$EnhancerByCGLIB$$f254165e' of reloadable type grails.plugin.jesque.JesqueConfigurationService is not reloadable: may not see changes reloaded in this hierarchy (please comment on that jira)
GRAILS-7799: Subtype 'grails.plugin.jesque.ScheduledJobDaoService$$EnhancerByCGLIB$$8f45c9d7' of reloadable type grails.plugin.jesque.ScheduledJobDaoService is not reloadable: may not see changes reloaded in this hierarchy (please comment on that jira)
GRAILS-7799: Subtype 'grails.plugin.jesque.TriggerDaoService$$EnhancerByCGLIB$$61a6907b' of reloadable type grails.plugin.jesque.TriggerDaoService is not reloadable: may not see changes reloaded in this hierarchy (please comment on that jira)
GRAILS-7799: Subtype 'grails.plugin.jesque.JesqueConfigurationService$$EnhancerByCGLIB$$f254165e' of reloadable type grails.plugin.jesque.JesqueConfigurationService is not reloadable: may not see changes reloaded in this hierarchy (please comment on that jira)
GRAILS-7799: Subtype 'grails.plugin.jesque.ScheduledJobDaoService$$EnhancerByCGLIB$$8f45c9d7' of reloadable type grails.plugin.jesque.ScheduledJobDaoService is not reloadable: may not see changes reloaded in this hierarchy (please comment on that jira)
GRAILS-7799: Subtype 'grails.plugin.jesque.TriggerDaoService$$EnhancerByCGLIB$$61a6907b' of reloadable type grails.plugin.jesque.TriggerDaoService is not reloadable: may not see changes reloaded in this hierarchy (please comment on that jira)
GRAILS-7799: Subtype 'grails.plugin.jesque.JesqueConfigurationService$$EnhancerByCGLIB$$f254165e' of reloadable type grails.plugin.jesque.JesqueConfigurationService is not reloadable: may not see changes reloaded in this hierarchy (please comment on that jira)
GRAILS-7799: Subtype 'grails.plugin.jesque.ScheduledJobDaoService$$EnhancerByCGLIB$$8f45c9d7' of reloadable type grails.plugin.jesque.ScheduledJobDaoService is not reloadable: may not see changes reloaded in this hierarchy (please comment on that jira)
GRAILS-7799: Subtype 'grails.plugin.jesque.TriggerDaoService$$EnhancerByCGLIB$$61a6907b' of reloadable type grails.plugin.jesque.TriggerDaoService is not reloadable: may not see changes reloaded in this hierarchy (please comment on that jira)
GRAILS-7799: Subtype 'grails.plugin.jesque.JesqueSchedulerThreadService$$EnhancerByCGLIB$$d60f53f9' of reloadable type grails.plugin.jesque.JesqueSchedulerThreadService is not reloadable: may not see changes reloaded in this hierarchy (please comment on that jira)
GRAILS-7799: Subtype 'grails.plugin.jesque.JesqueConfigurationService$$EnhancerByCGLIB$$f254165e' of reloadable type grails.plugin.jesque.JesqueConfigurationService is not reloadable: may not see changes reloaded in this hierarchy (please comment on that jira)
GRAILS-7799: Subtype 'grails.plugin.jesque.JesqueConfigurationService$$EnhancerByCGLIB$$f254165e' of reloadable type grails.plugin.jesque.JesqueConfigurationService is not reloadable: may not see changes reloaded in this hierarchy (please comment on that jira)

Workers die on exception recovery due to MissingPropertyException on exceptionHandlerRef access

Hi Michael,

Right now recoverFromException() attempts to access a private property from WorkerImpl superclass in Jesque causing all workers to die e.g. when connection to Redis is temporarily lost and then comes back online again ...

Error groovy.lang.MissingPropertyException: No such property: exceptionHandlerRef for class: grails.plugin.jesque.GrailsWorkerImpl

... can you pls. update the GrailsWorkerImpl.groovy with the following one-line change so that the public accessor is used when resolving the recovery strategy ...

// final RecoveryStrategy recoveryStrategy = this.exceptionHandlerRef.get().onException(this, e, curQueue)
final RecoveryStrategy recoveryStrategy = this.exceptionHandler.onException(this, e, curQueue)

Thanks.

Bean injection on autowired jobs via config

I found an interesting error when trying to use the redisService from within a job with using autowired workers from config. If I invoke a job directly with startWorker it works fine, but having the workers auto load from config appears to break injection/context of beans.

I have added a new test class to illustrate the issue. Build will be broken if running tests.

Update persistence mechanism

The persistence mechanism was copied from an old approach used in the grails background-job plugin. Update the mechanism to the newer approach in the executor plugin.

Made a mistake. Sorry!

Ability to configure redis database (number) for jesque

Thanks for the great product. It seems that at the moment it is not possible to change the default redis database (0) for jesque, since it does not try to read the database attribute from redis config. Can you please consider adding something like the following two lines to the doWithSpring closure in JesqueGrailsPlugin.groovy ...

if(redisConfigMap.database)
jesqueConfigBuilder = jesqueConfigBuilder.withDatabase(redisConfigMap.database as Integer)

... just before the def jesqueConfigInstance = jesqueConfigBuilder.build() call?

Jesque threads not robust to redis issues

The jesque worker threads and jesque scheduler thread can die if there are issues with redis without retrying. Additionally, some core jesque log messages seem to go to std out instead of the grails logging facility.

Upgrade to Jesque 1.3.0

I would love to modify the Grails Jesque plugin to expose some new features in Jesque 1.3.0. I attempted to upgrade the plugin myself, but an exception was thrown when the only change I made was upgrading the Jesque library.

Problem

I changed the version of Jesque in BuildConfig from 1.2.0 to 1.3.0.
Then I tried to run my Grails app that uses the Jesque plugin, and I got this exception:

java.lang.UnsupportedOperationException
at java.util.Collections$UnmodifiableMap.put(Collections.java:1342)
at grails.plugin.jesque.GrailsWorkerImpl.addJobType(GrailsWorkerImpl.groovy:47)
at net.greghaines.jesque.worker.WorkerImpl.setJobTypes(WorkerImpl.java:410)
at net.greghaines.jesque.worker.WorkerImpl.<init>(WorkerImpl.java:178)
at grails.plugin.jesque.GrailsWorkerImpl.<init>(GrailsWorkerImpl.groovy:24)
at grails.plugin.jesque.JesqueService.startWorker(JesqueService.groovy:115)
at grails.plugin.jesque.JesqueService$_startWorkersFromConfig_closure4_closure6.doCall(JesqueService.groovy:165)
at grails.plugin.jesque.JesqueService$_startWorkersFromConfig_closure4.doCall(JesqueService.groovy:164)
at grails.plugin.jesque.JesqueService.startWorkersFromConfig(JesqueService.groovy:159)
at JesqueGrailsPlugin$_closure5.doCall(JesqueGrailsPlugin.groovy:162)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)

I think I have diagnosed the problem, but I don't know what the fix should be.

Diagnosis

The key changes appears to be WorkerImpl in v 1.3.0 now calls addJobType, which it didn't do before. This ends up calling GrailsWorkerImpl.addJobType, which attempts to modify an unmodifiable map, which throws the exception.

Details:

It seems that in Jesque 1.3.0, the WorkerImpl's constructor is now calling the addJobType() method. It didn't do this in Jesque 1.2.0.
The GrailsWorkerImpl overrides this method with its own implementation.
The original implementation, in WorkerImpl, requires the jobType that is added to either implement Runnable or Callable.
Your implementation appears to be identical to the original, except it omits the Runnable/Callable requirement. This makes sense as the Job classes are POGOs.
Your implementation references "this.jobTypes"
The jobTypes member variable is declared private by WorkerImpl (and has always been private).
Therefore, "this.jobTypes" ends up calling the getJobTypes method.
The getJobTypes method returns an unmodifiable map.
In your implementation of addJobType, you call the put method on this unmodifiable map which results in the UnsupportedOperationException.

Has GrailsWorkerImpl.addJobType just not been getting called? It doesn't seem like this would have worked at any time in the past two years or so, which is as far back as I looked in the history for WorkerImpl. This map has been unmodifiable for a long time and the member variable has been private since the first commit.

Workaround

I did figure out a kind of ugly workaround.

Remove GrailsWorkerImpl.addJobType
Make all jobs implement Runnable, but with a run() method that only throws an UnsupportedOperationException, since we don't want this to actually ever get called and want to know if it does.
- All jobs in the plugin (AutoWireJob, DomainJob, etc) implement Runnable
- All jobs in my app

This did work. It has the added ugliness of worthless Runnable implementations on all of the jobs, but it seemed to work fine with Jesque 1.3.0.

Better Fix?

I am guessing you will have a more elegant, permanent fix for this.

If you can suggest something else to try, I will do it. Assuming it works, I can send a pull request to you with the upgrades.

Ability to set an exception handler to the worker?

Jesque allows to set a custom exception handler to the worker if one chooses to. I don't see a way of doing this in the plugin when the workers are configured externally and started automatically. Would it make sense to expose this as an option?

The reason is that currently the jesque-grails plugin works with the default exception handler. This exception handler does not have any logging and for JedisDataExceptions, the worker is terminated. The main problem with this pattern is that you don't really know if your worker has been terminated.