Comments (8)
I've come across this before I think. The camera receives args and kwargs already coerced into a string, so your options are to parse JSON-ish django string repr to actual JSON (what I'm doing at the moment), or change celery presumably quite fundamentally somewhere else so that the values arrive in the camera as JSON-encoded strings to begin with.
from django-celery-monitor.
Yes, that's what I've concluded/done too. Maybe close the issue?
from django-celery-monitor.
I just came across this, and I think args, kwargs and result should all be saved as JSON-encoded strings. I accept that some data cannot be properly handled, for those just stringify anything that causes json.JSONEncoder.default() to raise a TypeError.
I'd be happy to submit a PR for this if deemed acceptable.
from django-celery-monitor.
@ShaheedHaque That sounds fantastic to me if you were happy to do that. I'm not a project maintainer though, so would be good to hear from @jezdez if this was something he would merge. It might be good to add a flag so that anyone's existing parsing of these values is not borken.
from django-celery-monitor.
Indeed (and actually, I guess there is a conversation to be had about results, since AFAIK, that is allowed to be a non-JSON value, e.g. a bare int).
from django-celery-monitor.
Task arguments can be any valid Python type and will only be serialized with the configured task serializer when sent between Celery clients and workers.
The goal of this package is to use the task and worker event state to conduct monitoring, which in turn provides its values verbatim without serialization -- by design. So in other words we're erring on the side of correctness instead of convenience. There is also an additional operational risk of converting the arguments to JSON during storing that could lead to monitoring race conditions if for example the conversion to JSON fails and prevents updating the task state in the database.
There are a few options to get what you want nevertheless (with the caveat that you'd be on your own):
- subclass
django_celery_monitor.camera.Camera
, override theupdate_task
method (and calling the parentupdate_task
method first to continue the usual functionality) and store the arguments in JSON (or whatever form is convenient for you) in a separate datastorage (e.g. a separate data model) - we add a Django signal to this package (e.g.
celery_task_monitored
) so you can do option 1 without subclassing, the rest stays the same - post-process task state updates using Django's
post_save
signal and convert the arguments to the format you require, and store it in a separate table
from django-celery-monitor.
Thanks for the quick response. Is there a way to know, for a given event, what serializer was used? I don't see a content_type field in the model, for example?
from django-celery-monitor.
@jezdez I just added this debug into camera.py:
@@ -85,6 +85,7 @@
(task.worker.hostname, task.worker),
)
+ logger.warning('type(task.kwargs)={}: {}'.format(type(task.kwargs), task.kwargs))
defaults = {
'name': task.name,
'args': task.args,
And the resulting debug indicates that kwargs has, IIUC, already been coerced into a string even before being written to the TextField in the database:
2018-02-14 20:05:12,823 [WARNING] django_celery_monitor.camera: type(task.kwargs)=<class 'str'>: {'client': 8, 'company': 3, 'frequency': 'w1', 'next_T': '2018-10-07'}
2018-02-14 20:05:14,850 [WARNING] django_celery_monitor.camera: type(task.kwargs)=<class 'NoneType'>: None
2018-02-14 20:05:14,854 [WARNING] django_celery_monitor.camera: type(task.kwargs)=<class 'str'>: {'client': 8, 'company': 3, 'frequency': 'w1', 'next_T': '2018-10-07'}
2018-02-14 20:05:14,881 [WARNING] django_celery_monitor.camera: type(task.kwargs)=<class 'NoneType'>: None
Given that kwargs definitely started life as a dict, and what you confirmed about the intent being to use a loss-less on the wire format, this suggests that some unexpected string coercion is going on, right?
Also, given that the value is being stored in a database TextField, are we certain that the stored value would not be reduced to a string by virtue of being stored like this?
from django-celery-monitor.
Related Issues (20)
- Celery Monitor no longer works with Django 2.2 HOT 8
- Non default database
- Dependabot can't evaluate your Python dependency files
- Implement Jazzband guidelines for django-celery-monitor
- Is the project dead? HOT 6
- The configuration settings aren't working for me
- Incompatibility with Django 3.2.7 HOT 1
- How to set the Settings for the module ? HOT 1
- High CPU Usage HOT 3
- When using stopwait, tasks remain STARTED HOT 2
- taskstate can't save the result HOT 6
- Allow tags support has been removed in Django 2.0 HOT 2
- empty list of tasks in the monitor HOT 4
- Change color of "STARTED" string in the admin ?
- Name of expire variables in configuration HOT 2
- Master build failing due to network error?
- DoesNotExist('WorkerState matching query does not exist.',) HOT 5
- retries is always 0
- Add screenshot / "what do i get" overview HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from django-celery-monitor.