Comments (11)
The package currently does have support for tracking tasks in a group, although it does not have persistence as you've noted. An example view to show off group support would look as follows:
def task_view(request):
result = group(task.s(i) for i in [1000 for _ in range(5)])()
return render(request, 'http.html', context={'task_ids': [task for parents in result.children for task in parents.as_list()[::-1]]})
The task_ids
context will be used to get all of the tasks currently in the group, and the list comprehension is to simply act as a star expression to remove each task ID from the list that parents.as_list()
generates, and to reverse it so any children of that task are properly displayed as well (or else you'd have the first tasks at the bottom of the list!). An accompanying task would be:
@shared_task(bind=True)
def task(self, number):
progress_recorder = ProgressRecorder(self)
for i in range(number):
progress_recorder.set_progress(i+1, number)
return int(random()*1000)
For the html, the significant part of the page would have:
{% for task_id in task_ids %}
<div class='progress-wrapper-{{ forloop.counter0 }}'>
<div id='progress-bar-{{ forloop.counter0 }}' class='progress-bar-{{ forloop.counter0 }}' style="background-color: #68a9ef; width: 0%;"> </div>
<div id="progress-bar-message-{{ forloop.counter0 }}">Waiting for progress to start...</div>
<div id="progress-result-{{ forloop.counter0 }}"></div>
</div>
{% endfor %}
<script>
document.addEventListener("DOMContentLoaded", function () {
const task_urls = [
{% for task_id in task_ids %}"{% url 'celery_progress:task_status' task_id %}",
{% endfor %}];
for (var i = 0; i < task_urls.length; i++) {
CeleryProgressBar.initProgressBar(task_urls[i], {
progressBarId: "progress-bar-" + i,
progressBarMessageId: "progress-bar-message-" + i,
resultElementId: "progress-result-" + i
});
}
});
</script>
The for loop will ensure that each task ID is assigned it's own bar and result element, and when the page finishes loading, the JavaScript will start a bar instance to begin pulling data. This, in total, would produce something to the tune of the image provided below.
When the tasks finish, the end result will look like the image below.
If you have a callback that is supposed to activate after the task is finished, it would be fairly easy to add an if statement on the last task in the JavaScript for loop and modify the onSuccess to suit your needs. A native solution for this would be an interesting task to behold, but any suggestions are welcome!
from celery-progress.
@shared_task(bind=True)
def main_task(self, seconds):
progress_recorder = ProgressRecorder(self)
result = 0
for i in range(seconds):
time.sleep(1)
sub_task(seconds, progress_recorder)
result += i
progress_recorder.set_progress(i + 1, seconds)
return result
@shared_task(bind=True)
def sub_task(self, seconds, progress_recorder):
time.sleep(1)
task_status = progress_recorder.get_status()
progress_recorder.set_progress(task_status+1, seconds)
This is a quick rough idea. The reason for the need to be able to create subtasks that share a progress_recorder, is that if a group of tasks are to be executes async, it is a must that all iterations of the main task is created as their own tasks. As of right now i have gone with an implementation of django_celery_results.models.GroupResult. However this is only able to provide me with a count of completed vs total tasks in a group of tasks, i have not be able to use this with your library. This also mean that i am not able to track progress of the sub_tasks as i only track completion.
from celery-progress.
thanks - I'll take a look soon
from celery-progress.
I've never attempted this. A couple possible options that may work (just some quick thoughts):
- Show a separate progress per task using the normal observer
- Create some other observer object that can monitor all the child tasks and report progress on everything. Not sure whether you want to just track the number that have completed or the progress of each individual task in the group, but the latter would require passing information from the spawned tasks back up to the main thread somehow (sounds a bit tricky).
from celery-progress.
Hi @czue, I really appreciate your commitment :). I'm working with @petoor, that's why I'm commenting on this issue also. One of the issues with the current workflow for this project (if I understand correctly) is that the view is expected to return the task id so if multiple tasks are spawned from a group, there's no way to return the id's of all tasks. Is that right?
I'm working on #50 which I guess it would solve this by including the id of each subtask within a group in a table that links it to a particular user. Then this can be queried at any time.
I have currently a very simple implementation of #50 implemented that I'm testing at the moment. I'll be able to share it shortly in case it might be useful for this project.
from celery-progress.
Alternatively, if a single progress bar encompassing all tasks is what your after, that we currently don't support. As @czue noted, it would require passing information from all tasks into a centralized location. From there, this "super observer" would then have to try and figure out how "done" every task is, and I guess work out how to display those results. From what I can see from GroupResult
's source, it seems a GroupResult
is only returned once the group is started, which also is coincidentally the object that holds the information that would be used to get the task IDs. Alternatively, for a rather crazy idea, it would be interesting if each child task could be spawned with a "subserver" that will shout out it's status, with a "super observer" capturing this and doing generally the same as above. If either of these could be accomplished, it may be pretty useful.
from celery-progress.
hi could the main ProgressRecorder
be serialized somehow to the subtasks?
edit:
when i try to run the desrialized set_progress i get
[2021-03-08 18:59:39,920: ERROR/ForkPoolWorker-1] Task cluster123.tasks.task_progress_update[5e1e9d5a-54c6-4a89-b062-fcbbcbc283dd] raised unexpected: AttributeError("'dict' object has no attribute 'update_state'") Traceback (most recent call last): File "/home/arubico/PycharmProjects/djangoProject/venv/lib/python3.9/site-packages/celery/app/trace.py", line 405, in trace_task R = retval = fun(*args, **kwargs) File "/home/arubico/PycharmProjects/djangoProject/venv/lib/python3.9/site-packages/celery/app/trace.py", line 697, in __protected_call__ return self.run(*args, **kwargs) File "/home/arubico/PycharmProjects/djangoProject/cluster123/tasks.py", line 22, in task_progress_update prog_update.set_progress(i, t, m) File "/home/arubico/PycharmProjects/djangoProject/venv/lib/python3.9/site-packages/celery_progress/backend.py", line 50, in set_progress self.task.update_state( AttributeError: 'dict' object has no attribute 'update_state'
which is logical the deserialization isn't a constructor and this is a bad idea to begin with
from celery-progress.
Was this ever solved, I run the same problem as @safhac , trying to pass the ProgressRecorder from a main_task down to the sub_tasks it spawns, however this do not seem doable. Only solution ive come up with is for the main task to save the sub_task id´s which are generated when spawning them. And then building a system to iterate over the list of id´s and calculate a progress.
from celery-progress.
Apologies for the radio-silence. We can likely add serialization to the ProgressRecorder
class if that's all it takes? If someone who has run can provide a dummy example of the code that uses groups and/or spawns subtasks I'd be happy to try and get it working and then submit an updated version with support for this. Alternatively, I'd happily accept contributions on this.
from celery-progress.
I was able to get something working. See #115. Would love any feedback/testing as I didn't explore the edge cases.
Note, I explicitly used GroupResult
instead of working with this example. @WassawRoki does this work for your use case?
from celery-progress.
Interesting, i will look in to this implementation asap :D
from celery-progress.
Related Issues (20)
- Using master branch leads to Javascript TypeError HOT 1
- DOM only updating last element of the loop while diplaying progress HOT 2
- django.conf.urls.url() deprecated in Django 3+, removed in 4.0 HOT 3
- using websockets with channels_redis 4.0.0+ leads to RuntimeError("Event loop is closed") HOT 2
- NoReverseMatch at /status 'celery_progress' is not a registered namespace HOT 2
- celery Progress on file upload HOT 1
- Get the actual value when onProgress HOT 4
- Update two bars relatively to two workers HOT 6
- Maintenance and support for Django >= 3.1 HOT 3
- Race between creating ProgressRecorder and setting its progress HOT 6
- Introduce changelog HOT 3
- get_progress endpoint allows retrieval of arbitrary tasks HOT 7
- How would one use this progress bar when a task is wrapped in a Django transaction.on_commit? HOT 4
- Create Sdist HOT 9
- Handling a FileResponse HOT 3
- Config Mensages HOT 1
- 'celery_progress' is not a registered namespace
- ajax request using htmx HOT 1
- This could benefit from documantation for how to use it with chord or group based workflows. HOT 4
- Legacy install method will be deprecated soon HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from celery-progress.