Comments (6)
Hey @morenol! That would be great. I think the steps to tackle this are as follows:
- Start by designing a
Backend
trait, and show how theCelery
app would interact with it. You could submit this as a PR or even just post a code-block in this thread with some explanation. - Figure out how Python's Celery implements the Redis backend under the hood. In particular, we need to know the serialization format and which Redis functionality is used so that we can be compatible with the Python implementation. This might be the hardest part since the internals of Python Celery are not well documented.
- Implement the
RedisBackend
based on what we've found in Step 2. - Add unit tests and integration tests.
- Bonus points: add an integration test that checks cross compatibility with the Python implementation.
from rusty-celery.
That sounds like a good plan!
from rusty-celery.
Hi folks! For anyone interested, I implemented a very quick and dirty persistence backend for the project I'm working on. The code is under this task_records module:
- types – The
TaskRecord
type, which records a task execution result. - redis_impl –
RedisTaskRecords
, which implements saving and retrievingTaskRecord
values, and provides a helper to send tasks and retrieve their results as one action. - callbacks – Task
on_success
/on_failure
callbacks to save results usingRedisTaskRecords
.
This implementation only covers our project's most immediate needs, so it has a lot of limitations and cautions, but it might provide a useful reference point for further work on this issue.
from rusty-celery.
I would like to take this, do you have any thoughts, suggestions, guideline that I should consider before jumping into it?
from rusty-celery.
I have been working on this somewhat as I would personally find it useful. AsyncResult still needs altering to work with backends, AsyncBackend and ResultConsumer are also not finished. The parallel to Message/Delivery for backends still needs to be settled. And lastly, none of the actual hooking up of backends in the main app has been finished nor have tests been written.
Despite the work remaining, a code review would be very helpful at this point as well as some architectural discussion. If anyone is willing to engage I will drop a PR. As of now it compiles and all of the previous functionality works despite any changes. Test cases are also giving same results as before.
If there is interest, I will compose a more detailed write up of the work alongside the PR. Otherwise I will just keep hacking along on my own. 👍
from rusty-celery.
This PR is awesome. Any way we can complete the review and merge it?
from rusty-celery.
Related Issues (20)
- rustls support
- Add task options that enable the user to send negative acknowledgements and rely on broker's retry mechanisms HOT 2
- Prefetch not working properly
- No connection was performed when setting `broker_connection_retry` to `false`
- How do I run the main method in beat_app.rs in a thread
- Cannot connect to broker with x-max-priority set HOT 2
- How to get TaskResult from AsyncResult? HOT 1
- New release? HOT 3
- Call for new features HOT 5
- BrokerBuilder is not Send HOT 6
- Socket was readable but we read 0. This usually means that the connection is half closed this mark it as broken HOT 7
- Celery task blocking main thread in actix web HOT 3
- does the RedisBroker support password
- WARN celery::broker::redis:Setting heartbeat on redis broker has no effect on anything HOT 2
- support streams protocol
- is it possible to define the log output level HOT 1
- Postgres as a broker/backend
- Limiting the rate of consuming HOT 4
- Delivery acknowledgement timeout on best practices
- Flower integration HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rusty-celery.