requests.exceptions.ReadTimeout about dispatcher-container HOT 9 CLOSED

the reason seems to be that the request is too large.
Something in the underlying packages changed to decrease the limiting size - for safety. I released the limit.

https://github.com/oda-hub/dispatcher-container/blob/master/entrypoint.sh#L30

this seems to help retrieving the mosaics. I did not get them all.

however, large requests will end up being a problem occasionally. We should send large scw lists by other means, with file or URL (the latter option will also help making much shorter request URLs, removing this "impossible to make URL"). oda-hub/oda_api#98 , oda-hub/dispatcher-app#301

from dispatcher-container.

With the understandable limit of 500 scw, if I need to get a product over the full mission lifetime, I cannot use time periods, as the number of available science windows is variable with time. I need to use a science window list, in the past, I tested with 50 scw, but now I exploit 300. It crashed again just now with the very same error.e

from dispatcher-container.

The fact that you see the very same error tells very little. Too many causes lead to timeout. Including client-side.
Now, there is no limit on the request size, so the cause of the timeout is different. And happens more rarely.

There is no single cause, but there are some parts which take longer with large requests. Most can be improved similarly easily. I can address several places quickly, tomorrow if nothing else comes up.

Meanwhile the client should retry (oda-hub/oda_api#99 you can try this https://github.com/oda-hub/oda_api/tree/safe-retry). Timeout is usually recoverable.

Even without extra retries, I was able to get all of the mosaics you requested (including quite some with OSA10, apparently, see the incident on why).
Then, it takes very long to merge them though. I merged a subset (it takes some hour it seems).

from dispatcher-container.

Do I have to update oda_api to get the products ?

from dispatcher-container.

Even without extra retries, I was able to get all of the mosaics you requested (including quite some with OSA10, apparently, see the incident on why).
I did not find the incident on OSA10 in the issues

from dispatcher-container.

Do I have to update oda_api to get the products ?

no, since I did not need to. but it might help in some cases.

from dispatcher-container.

Even without extra retries, I was able to get all of the mosaics you requested (including quite some with OSA10, apparently, see the incident on why).
I did not find the incident on OSA10 in the issues

this incident:

https://mattermost.astro.unige.ch/cdci/channels/jobs-of-10k-scw-total-just-keep-running

at least part of the these were submitted with your notebook but by myself: since an older wrapper version substitutes internally osa version. It was not explicitly required by the notebook.

I added this

import pkg_resources
from packaging import version

assert version.parse(pkg_resources.get_distribution("oda_integral_wrapper").version) >= version.parse('1.3.25')

Also, I noticed there were other scws computed than those in the notebook I got. Notably from other time periods (I tried to expand time period to get some new running jobs, but the images were already there)

Since the upgrade discussed above, but without oda_api upgrade, I saw the timeout couple of times, at random moments. Infrequently enough to fetch 30 successfully. I did not even manage to meaningfully try the oda_api upgrade live - the problem does not re-occur for me.

I suggest you try with the upgrade anyway:

$ pip install git+https://github.com/oda-hub/oda_api@safe-retry --upgrade --ignore-installed

Even if timeout happens, it will not break the loop, and it will succeed after some tries.

We should also more actively notice user having these issues, though it is not always so trivial.

from dispatcher-container.

to check again, I re-ran them all, while also running other requests:

https://gitlab.astro.unige.ch/reproducible_INTEGRAL_analyses/igrj17315/-/blob/4a6f842f2e70d8798ca5316a2d053f9727387e89/igrj17315.ipynb

from dispatcher-container.

I note that now, underload, I can see the retries at last. I will see to reduce them. But improved oda-api seems to work ok

from dispatcher-container.