Comments (3)
I accomplished a behavior similar to this by queuing the second job at the end of the first job. i.e. I run a script as my first job and at the end of the script I added some logic to enqueue a new job depending on the outputs that were just created.
from rq.
I ended up doing something similar to that, where I read the contents of the queue to see what jobs are running or scheduled, then add the new job to depend_on the last job in that list.
I was hoping to find a more elegant solution, though.
My code looks something like this:
queue = 'the selected queue'
job_id = 'some string'
q = Queue(queue, connection=redis)
from rq.registry import StartedJobRegistry, DeferredJobRegistry
running_jobs = StartedJobRegistry(queue, connection=redis).get_job_ids()
deferred_jobs = DeferredJobRegistry(queue, connection=redis).get_job_ids()
with app.app_context():
if len(deferred_jobs) > 0:
last_job = deferred_jobs[-1]
job = q.enqueue(f=args[0], job_timeout='1h', job_id=job_id, args=args[1:], depends_on=last_job)
elif len(running_jobs) > 0:
last_job = running_jobs[-1]
job = q.enqueue(f=args[0], job_timeout='1h', job_id=job_id, args=args[1:], depends_on=last_job)
else:
job = q.enqueue(f=args[0], job_timeout='1h', job_id=job_id, args=args[1:])
from rq.
It sounds like you're using Queue as individualistic rather than a collection
[q1, q2, q3] is analogous to [high-prioritiy-queue, medium-priority-queue, low-priority-queue] within the queue namespace redis:queue:
Which is why you're seeing the behavior regarding job/function ordering
Given your use case, I would take a look at Queue(redis_queue_namespace_prefix: str)
. This would allow you to create a queue namespace per customer to allow for non-overlapping Jobs.
https://github.com/rq/rq/blob/a8209391fffe3bad195264e6ed22724e36ae1fb5/rq/queue.py#L69C1-L69C52
customer_foo: 'foo'
customer_bar: 'bar'
foo_q = Queue(['q1', 'q2', 'q3'], connection=redis, redis_queue_namespace_prefix=f'redis:queue:{customer_foo}:')
bar_q = Queue(['q1', 'q2', 'q3'], connection=redis, redis_queue_namespace_prefix=f'redis:queue:{customer_bar}:')
foo_q.enqueue(my_func_1)
bar_q.enqueue(my_func_1)
foo_q.enqueue(my_func_2)
Depending how you manage your workers, that might not be feasible.
Whether or not it is, If you aren't using dependencies already, I would recommend it.
Line 1537 in a820939
There seems to be some code that allows for dependencies to be
enqueued_at_front
that could potentially resolve the problemLine 67 in a820939
Good luck!
from rq.
Related Issues (20)
- Replace Redis with open source alternative? HOT 5
- Questions about Copilot + Open Source Software Hierarchy
- How do I log with INFO level from a job?
- no way to catch the redis.exceptions.ConnectionError on pubsub.get_message HOT 4
- Integrating OpenCV with FastAPI and Redis RQ Causes Task/Job Failures HOT 4
- Installing setproctitle >= 1.3.0 breaks RQ on MacOS HOT 1
- Job B depends on job A. A fails. Is there a parameter set that would result in B marked as failed?
- Using `@job` decorator and `func.delay()` causes subclass classmethods to not work correctly HOT 1
- Periodic High CPU with RQ 1.16.1 (not in RQ 1.12.0) HOT 1
- Orphaned "queued" job after shutdown/restart of worker
- [Function Request] Hope to support scheduler cron syntax HOT 3
- [bug] --with-scheduler The scheduled task with the queued name cannot be executed.
- Run Scheduler Immediately Upon `rqworker --with-scheduler` HOT 2
- Error "signal only works in main thread of the main interpreter" after a system failure HOT 1
- The @job decorator is not working as documented
- `import_attribute(...)` in `utils.py` swallows import errors in custom worker classes when running via the CLI. HOT 1
- [feature] WorkerPool with different configuration for queues HOT 2
- Use typing.Protocol to strictly enforce callback function signatures
- How to check why a job is killed? HOT 1
- Admin privileges required to boot RQ Worker
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rq.