Hi @czue ,
Although the celery progress-bar works fine for me. I am setting set_progress(start,end). End is not 100.
Even though before tracing the progress the UI shows 0 of 100 processed until the progress starts in backend. Once the process starts in backend it changes to my desired value.

Hi guys,
I would like that you consider make de messages like 'Task started...' configurable, to permite translate this.

Regards,

Recently when we tried deploying our app and installing all the packages, ran into an issue:

django-celery 3.2.2 depends on celery<4.0 and >=3.1.15

Seems like there has not been much update to django-celery and celery 5.0 has been released.
Django-celery is needed to run celery progress bars.

Downgrading celery to 3.1 is not a choice as that is a very old version and old packages like kombu also need to be downgraded.

Any suggestions how to proceed ?

Hi @czue, I have tried your advice but the error still there. In my case, the webpage was worked but when I upload a video, the web shows me the error. It is said the same look as the post of @ludwigwittgenstein2. Could you help me, please? Hope to hear from you soon.

Thank you so much.
Best regards.

Does this package can also be used in flask instead of Django?

I use Flask-Celery-redis.

this does not happen if i use CELERY_RESULT_BACKEND == redis.
The status of the task while processing appears as so:

{"complete": false, "success": null, "progress": {"pending": true, "current": 0, "total": 100, "percent": 0}}

once completed though, all values are correct:

{"complete": true, "success": true, "progress": {"current": 100, "total": 100, "percent": 100}, "result": {<VALID_RESULTS>}}

Thank you!

Any user, independent whether they are logged in or permitted, can get data about any task using the get_progress endpoint.

Would be nice if there was support for using websockets instead of polling.

Currently, there seems to be two task states that will slip past the currently accounted for states in Progress.get_info():

1. IGNORED
   • HTTP: When used with a progress recorder's stop_task() as demonstrated in #4, stop_task() will function as expected. However, if stop_task() is not used, upon finishing the task, the result backend will still have the PROGRESS state, meaning the client-side javascript will continue to request the same resource over and over again until the page is closed.
   • WS: As with HTTP, when used with stop_task(), it works fine. Without, the progress bar will get to max but never complete, and as the task has already technically finished, no further updates will be sent. The JS will be left to sit, waiting for a response that will never come, and will wait until the server times out the connection.
2. RETRY (as noted by #53)
   • HTTP: Once the retry exception is thrown, the progress request will try to serialize the error and fail, returning a 500 error for the http request and killing the bar.
   • WS: Once the retry exception is thrown, the progress request will try to serialize the error and fail, forcing the post-run handler to retry sending the error over and over until the retried task is successful.

Both situations pose an interesting challenge on how they'll be handled. For IGNORED, it may be worth revisiting stop_task()'s implementation and offering a possibly better solution than what is currently being used. Successfully fixing the handling for RETRY could eventually pave the way for #13 to get some as love as well. Thoughts on this would be greatly appreciated!

Reviewing changes of this package while updating dependencies is very hard because there is no changelog. It would be great to introduce it to get more transparency.

Is there any way or a recommended way of tracking long term jobs? As in keeping them in a table for querying afterwards?

In my case I'm dealing with jobs that might take 1-2 hours and if the user leaves the page during that time I don't have access to the job id returned by the view. I was planning on creating a table for keeping track of jobs.

Any suggestions on this? Or anyone dealt with a similar situation? I would be willing to contribute if this is something that might interest other people.

If a ProgressRecorder is created, its state is not initialized until set_progress() is called. During this time, I think there is a race condition where a page calling initProgressBar() can get a bad state === null and fall into onTaskError() during this time.
(progressBarMessageElement will say "Uh-Oh, something went wrong! Unknown state")

As a result of this change in #31, an exception triggered in a task will appear in both the result and progress bar message elements using the default configuration. Running a task with the code

@shared_task(bind=True)
def http_error_task(self, number):
    progress_recorder = ProgressRecorder(self)
    for i in range(number):
        time.sleep(.1)
        progress_recorder.set_progress(i+1, number)
        if i == int(number/2):
            raise StopIteration('We broke it!')
    return random()*1000

produces a progress bar as shown.
It seems counter-intuitive to me to include the exception message in two places, especially since the result makes more sense in the result element than the progress bar area. That's my personal take on it, however. I'd be happy to hear your thoughts on this.

Retrying tasks causes an error.

I think it's because Progress.get_info reaches the last line and returns an exception. The view tries to send it as JSON but can't.

Example task:

def retry_task():
    try:
        raise StopIteration('We broke it!')
    except StopIteration as exc:
        raise self.retry(exc=exc)

Hi @czue ,

Everything seems to doing well but for an unknown reason, i can't see my progress bar evolve in the same time has the info from the task_id url.

I see the progress bar going directly to "success". Looks that the Fetch function return only when the task has finished.

I have followed all the steps from the README

Any known reason ?

Thanks !

This piece of warning comes up when using Django 3.1

celery_progress/urls.py:6: RemovedInDjango40Warning: django.conf.urls.url() is deprecated in favor of django.urls.re_path().
  url(r'^(?P<task_id>[\w-]+)/$', views.get_progress, name='task_status')

That should be corrected in the next release, I suppose.

I updated one of my application from 0.0.10 to 0.0.14 and now running into issue where my custom onSuccess function is not able to access the result content. Here is a snapshot of it:

function processSuccess(progressBarElement, progressBarMessageElement, result) {
		console.log('SUCCESS: ' + result)
		var search_id = result[0]
		var datasheet = result[1]
		var payload = {
			"search_id" : search_id,
			"datasheet" : datasheet,
		}
		console.log(payload)

                ....
}

(result is now empty).

And here is the task:

@shared_task(bind=True)
def ProcessDownload(self, manufacturer, part_number, search_id):
	progress_recorder = ProgressRecorder(self)
	progress_recorder.set_progress(0, 10, description='Initializing Request')
	time.sleep(1)

	spec = Specifications(task=self)
	spec.DownloadPDF(manufacturer, part_number)

	if not spec.datasheet:
		return [search_id, '']
	
	return [search_id, capspec.datasheet]

This code used to work and the processSuccess would capture the content of result but no anymore... I've added those debug prints to make sure the processSuccess method is called (which is the case).
I've also changed the task return value to a simple string for testing purpose and still nothing.

Any idea what am I missing? Was this setup wrong in the first place?
How should one efficiently implement the "communication" between the task return values and the result input of the onSuccess method?

Hi, I'm trying to install a progress bar functionnality, and I'm new to Celery and asynchronous tasks.

Initially, the use of celery-progress seemed quite straightforwards
... until I realized I needed to install and configure Celery

There things started to get a lot more complicated as I tried to follow the link you provided in the close issue #29

You seemed to have done quite a good job with celery-progress to get things smoother, so it's a bit of a downer that the start-up of it kind of kills the initial enthusiasm.

Would it be possible to provide a simple github demo so that people can easily figure out what needs to be implemented 'in real life' to get celery-progress working?

Or if some info is already availlable somewhere, could you provide a link ?

I have followed the README to include all the original suggested code to try and get a progress bar to display. However, by using the print-technique I notice that my view function can never render the intended HTML file because it gets stuck on the mytask.delay(10) call. Do you know why this would be happening? Is there anything else not included in the README that needs to be added to the code? Additionally, this is my first time ever using an asynchronous task. Thanks!

It seems the progress bar will not update anymore when it encountered a network Error.
For example, when I flap the network, the progress bar will stop updating forever.
I also tried to install celery-progress[websockets,redis] instead of celery-progress, but it still does not work.
Do you have any good idea to make the progress bar auto-update when the network is restored?

I actively use this module and it works great. Is there a way you could enhance the UI of the celery progress bar from just a flat blob increasing every percent to a more professional progress bar with a GIF !?

Hello,

I am looking for a way to issue a redirect if the task completes as success or failure within the JavaScript itself.

I have the status bar working, and it's awesome. However, I have a list of 2 separate tasks. I want to submit one task, wait for the completion while viewing this nice status bar, then once complete (if successful) kickstart the second task and wait for that nice status bar to finish.

when I use ProgressRecorder the progress bar updates at 100% and even the custom function is called but when I use WebsocketProgressRecorder with channels==2.2.0 and redis at reaches 100% and stays their. on further debug the status remains (in event in js) remains pending.

I got this problem on my side. The get_progress views failed all the times because of Decimal object in the result returned by progress.get_info().

In my fork I did this update to handle the error. Would be great if you can review your main version. Thank you for this great package btw, saved tons of time for me :)

Hello,
Could you add never_cache decorator to get_progress view ? This will avoid having bar loading problems when UpdateCacheMiddleware is activated.
Thanks,
Martin.

It seems that this modules does not properly sync in a production-like deployment of django with uwsgi+nginx. It works fine in our test env. of our automation tool running the django webserver, but in production where we have uwsgi+nginx, it doesn't show the progress.
It gets stuck at "Waiting for progress to start.."

Hello.

Is there any way to track the progress of a celery group?
That is, spawning a lot of asynchronous tasks, and keeping track of how many subtasks are completed as a function
The celery group has a completed_count() option which does exactly that, but my understanding of celery-progress is not good enough to know if this can be incorporated
https://docs.celeryproject.org/en/latest/reference/celery.result.html#celery.result.GroupResult

if our group code looks something like this
group(do_something_to_file.si(file) for file in file_list)
then i'm not sure where to put the observer, since every subtask has a unique task id.
We could also assign an id to the group itself, but then again im not sure where to put the observer.

Best regards.
Peter

Getting the following error when deploying:

NoReverseMatch at /fv/
Reverse for 'task_status' with arguments '('',)' not found. 1 pattern(s) tried: ['celery-progress/(?P<task_id>[\w-]+)/$']

I use the following to display the progress bar, and the page refreshes constantly during the "crawl" of the progress bar and then continues to refresh indefinitely after the Celery task has completed. Perhaps my syntax is wrong?

        <script src="{% static 'celery_progress/celery_progress.js' %}"></script>
        <script>
            $(function () {
                var progressUrl = "{% url 'celery_progress:task_status' task_id %}"
                CeleryProgressBar.initProgressBar(progressUrl, {
                    onSuccess: $(function () {
                        location.reload();
                    })
                });
            });
        </script>

I am using django-celery-results as well, if that matters.

Thank you for the hard work on this application!

I am not that familiar with celery and don't even know if it can run multiple tasks concurrently, but if it does, would there be a way to support multiple loading bars that show the progress of each task?

It would be more robust if the javascript code on the front-end part (re)tried again after an XMLHttpRequest() failure.

That'd be useful in particuler when running Django's built-in webserver on http://localhost:8000 (using manage.py runserver) because when in development mode (DEBUG=1) Django's builtin webserver reloads python source files that change by restarting the webserver and the short unavailability of the webserver makes the progressbar stop (I'm using the default refresh rate of 500ms).

Thank you.

I did see someone did ask about doing a redirect on a task complete and had no error/was successful. I am after something similar but looking to redirect to a url from my urls.py not too familiar with javascript so trying to figure out exactly how I would go about this. I have got my template setup to run the progress bar just fine. But then my goal would be to after 'x' amount of seconds (undecided yet) to redirect to a url as an example for the current work I am looking to redirect to this url:

path('search/<int:pk>', views.search_list, name='search_list')

For now I am struggling as the view the progress bar is being called from wil be using the return render:

return render(request, 'prop/search_multiple.html', {'formset': formset,'task_id': task_id})

In the template I have the progress bar:

{% if task_id %}
        <script type="text/javascript">
            // Progress Bar (JQuery)
            $(function () {
                var progressUrl = "{% url 'celery_progress:task_status' task_id %}";
                CeleryProgressBar.initProgressBar(progressUrl, {})
            });
        </script>
    {% endif %}

Would I be adding something into to do with onsuccess? Or would I add something elsewhere? I really need to get around to learning some javascript so I have have a go at these myself but I thought asking on here could be a good way about it. I am not exactly sure if this is the place I should be posting this sort of question either as a bit new to how people use Github and if this is generally used more for actual bugs in the code or suggestions for it.

At https://pypi.org/project/celery-progress/#files there is no sdist, as is standard pypi/python practice.

Any plans to create one? It would greatly simplify the work of your downstream packagers.

Thank you.

i get the following error

SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data

when i use the default code from the readme

When I run my webpage in django, I am getting the error 'module' object has no attribute 'my_task'. I am importing it properly I hope in my views.py.

from standalone import tasks
result = tasks.my_task.delay(10)

I have a Django function that generates a csv or json file from database queries and returns a FileResponse, which opens a system File/Save dialog on completion. This works ok, except that some requests can take a while (10-30 seconds), so I want Celery to manage this and provide a progress bar. The return looks like this in a download_file() function -- after writing a file fn:

response = FileResponse(open(fn, 'rb'), content_type='text/json')
response['Content-Disposition'] = 'attachment; filename="'+os.path.basename(fn)+'"'
return response

I use Celery for other long tasks in the app, so I'm reasonably familiar with it. None of those return files though--they perform some tasks to retrieve data and write it to the database, and return only a JSON object summarizing results.

Using your library, I've made download_file a @task, initiated it with form submission for parameters, and it generates the file as expected, but Celery errors out with
EncodeError: Object of type FileResponse is not JSON serializable

I saw a suggestion to wrap the file in ContentFile() like so
response = FileResponse(ContentFile(open(fn, 'rb')), content_type='text/json')
The file still gets created, but the Celery error is now
a bytes-like object is required, not '_io.BufferedReader'

All this means that I can't get the File/Save dialog to open on completion. After lots of searching, I've found nothing about Celery handling Django FileResponse -- any ideas?

This is usage question. I'm unclear how to handle errors that I need to show in the progress bar.

If I raise an error for a celery task, then the celery-progress view's "result" field is null. If a handle an error in a celery task and return the error string, then the celery-progress view rightly populates its "result" field with the returned error, but obviously this is a "false" success. This leaves me wondering how I should correctly pass celery task errors back to the progress bar in a customized onError function?

When using the websocket progress reporter on my Linux production machine, all tasks follow up with

[2019-10-28 01:58:04,928: ERROR/ForkPoolWorker-1] Signal handler <function task_postrun_handler at 0x7fe145e94bf8> raised: RuntimeError('Never call result.get() within a task!\nSee http://docs.celeryq.org/en/latest/userguide/tasks.html#task-synchronous-subtasks\n',)
Traceback (most recent call last):
  File "/var/www/staging-logs/venv/lib/python3.6/site-packages/celery/utils/dispatch/signal.py", line 288, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/var/www/staging-logs/venv/lib/python3.6/site-packages/celery_progress/tasks.py", line 12, in task_postrun_handler
    WebSocketProgressRecorder.push_update(task_id)
  File "/var/www/staging-logs/venv/lib/python3.6/site-packages/celery_progress/backend.py", line 78, in push_update
    {'type': 'update_task_progress', 'data': {**Progress(task_id).get_info()}}
  File "/var/www/staging-logs/venv/lib/python3.6/site-packages/celery_progress/backend.py", line 111, in get_info
    'result': self.result.get(self.task_id) if success else None,
  File "/var/www/staging-logs/venv/lib/python3.6/site-packages/celery/result.py", line 205, in get
    assert_will_not_block()
  File "/var/www/staging-logs/venv/lib/python3.6/site-packages/celery/result.py", line 41, in assert_will_not_block
    raise RuntimeError(E_WOULDBLOCK)
RuntimeError: Never call result.get() within a task!
See http://docs.celeryq.org/en/latest/userguide/tasks.html#task-synchronous-subtasks

I find this especially odd because the error does not occur on my development Windows machine. I seem to have located a fix here. I will open a PR if this does indeed fix my issue.

Since it's inception, stop_task (in my opinion) has sat in a rather weird place:

• It has current and total arguments, neither of which matter because the progress is discarded for the default 100% once it "errors".
• The example use case shows catching an error, sending task metadata that the task has failed, and then ignoring the result
  • Is this not the same as simply letting the error propagate to the frontend?
  • If you have a desire to have a custom exception message, why not simply raise Exception(exception_message_string) or the like instead?
• stop_task doesn't actually stop the task on it's own, you HAVE to raise an exception anyways for it to stop. All stop_task does is tell any currently running bars to stop listening.
• An edge case presents itself in websocket progress bars if the task "failure" message fails to get through. The post-run handler will only receive information related to the "Ignored result", and will cause any progress bars that are listening to display an incorrect error if it fails to properly disconnect from the stop_task call (see #54).
  • This can be fixed with making the Ignore exception raise inside of stop_task, and pass it metadata that the post-run handler can try to decipher whether the Ignore exception means "Ignore", or "We actually mean FAILURE but we'd rather do it the hard way instead". This would require having to type-check every result that passes through the handler, and if it's an Ignore exception, look to see if specific metadata is present to then change the state and result to send to the progress bar. However, what's the point of doing that vs. actually just raising the error in the first place?

stop_task may have been a great idea at the time, but I feel that it's rather antiquated for the reasons listed above. Obviously, this would be a breaking change, so that must be taken into consideration. @czue @OmarWKH thoughts?

Websocket breaks when using django celery results because of django ORM.

Hi, thanks for this library,

Can anyone please demonstrate how to use this in a react project with Django/Django Rest Framework backend?

Related to #56.

Websocket breaks when channels 3 is installed. Channels 2.4.0 works.

Your usage notes for the view depends on the task returning a task_id:

def progress_view(request):
    result = my_task.delay(10)
    return render(request, 'display_progress.html', context={'task_id': result.task_id})

but a transaction.on_commit does not return a task_id, since the task may not have started yet. Any suggestions on how to use your progress bar with transactions?

Thanks!
Mark

from django.utils.module_loading import import_by_path
ImportError: cannot import name 'import_by_path'

Is there a workaround for this yet? import_by_path was deprecated in django 1.9
I'm using django 2.0.6

Is there a way to de-initialize the celery progress bar after everything is completed? My server constantly shows:
[09/Sep/2019 21:20:23] "GET /celery_progress/task-id-stub/ HTTP/1.1" 200 54
[09/Sep/2019 21:20:24] "GET /celery_progress/task-id-stub/ HTTP/1.1" 200 54
[09/Sep/2019 21:20:25] "GET /celery_progress/task-id-stub/ HTTP/1.1" 200 54

I tried CeleryProgressBar.initProgressBar(''); but did not work.

In the quickstart example in the documentation, the javascript code is (here for JQuery):

// JQuery
var progressUrl = "{% url 'celery_progress:task_status' task_id %}";
$(function () {
  CeleryProgressBar.initProgressBar(progressUrl)
});

I wanted to have multiple progress bars and used this in a Django template in a loop and it took me some time to recognize that this does not work: The variable progressUrl will be overwritten in each loop iteration and at the end there is only one URL and many progress bars referring to it.

My suggestion: Move the variable progressUrl into the function which is called when the document is ready:

// JQuery
$(function () {
  var progressUrl = "{% url 'celery_progress:task_status' task_id %}";
  CeleryProgressBar.initProgressBar(progressUrl);
});

In my use case this difference is important.

I could prepare a PR and also fix the missing return from #10. Are you fine with this, @czue ?