hynek / stamina Goto Github PK
View Code? Open in Web Editor NEWProduction-grade retries for Python
Home Page: https://stamina.hynek.me/
License: MIT License
Production-grade retries for Python
Home Page: https://stamina.hynek.me/
License: MIT License
I'd like to log retries similar to what is done in the tenacity documentation.
https://tenacity.readthedocs.io/en/latest/#before-and-after-retry-and-logging
At current work, we use datadog, not prometheus. At previous work, we used an internal metrics tool.
Since the _instrumentation
needs are pretty modest, how would you feel about defining an "observation sink" protocol and having a way for stamina
to be configured with a dynamic callable or module to receive the observable event?
I see the implementation already uses an INSTRUMENTS
list, so I think this might just mean stabilizing the interface and documenting how to add to INSTRUMENTS
?
I'm seeing some strange behavior from stamina, where a retriable exception is raised before the max attempts is reached.
Python: 3.10.4
stamina: 24.2.0
Sentry reports this log before the exception is raised:
stamina.retry_scheduled
{
stamina.args: [],
stamina.callable: <context block>,
stamina.caused_by: ReadTimeout(''),
stamina.kwargs: {},
stamina.retry_num: 3,
stamina.wait_for: 0.69,
stamina.waited_so_far: 1.64
}
stamina.retry_num
is 3, but attempts
is set to 10 in retry_context
Code:
async def handle_async_request(self, request: httpx.Request) -> httpx.Response:
for attempt in stamina.retry_context(
on=(
httpx.RemoteProtocolError,
httpx.ReadError,
httpx.ReadTimeout,
),
attempts=settings.HTTP_CLIENT_RETRIES,
):
with attempt:
return await self._wrapper.handle_async_request(request)
# This code should be unreachable, since stamina will raise an exception
# if the maximum number of retries is exceeded.
raise Exception("Max retries exceeded.") # pragma: no cover
Stack trace:
ReadTimeout: null
File "httpx/_transports/default.py", line 69, in map_httpcore_exceptions
yield
File "httpx/_transports/default.py", line 373, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "httpcore/_async/connection_pool.py", line 216, in handle_async_request
raise exc from None
File "httpcore/_async/connection_pool.py", line 196, in handle_async_request
response = await connection.handle_async_request(
File "httpcore/_async/connection.py", line 101, in handle_async_request
return await self._connection.handle_async_request(request)
File "httpcore/_async/http11.py", line 143, in handle_async_request
raise exc
File "httpcore/_async/http11.py", line 113, in handle_async_request
) = await self._receive_response_headers(**kwargs)
File "httpcore/_async/http11.py", line 186, in _receive_response_headers
event = await self._receive_event(timeout=timeout)
File "httpcore/_async/http11.py", line 224, in _receive_event
data = await self._network_stream.read(
File "httpcore/_backends/anyio.py", line 31, in read
with map_exceptions(exc_map):
File "contextlib.py", line 153, in __exit__
self.gen.throw(typ, value, traceback)
File "httpcore/_exceptions.py", line 14, in map_exceptions
raise to_exc(exc) from exc
ReadTimeout: null
File "app/services/external/gamification.py", line 25, in get_entity_for_action
entity_response = await get_blocks_pivot_data(field=EBlockField.ID, value=action.entity_id)
File "app/http/courses.py", line 243, in get_blocks_pivot_data
response = await pool.courses.get(
File "httpx/_client.py", line 1801, in get
return await self.request(
File "httpx/_client.py", line 1574, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
File "httpx/_client.py", line 1661, in send
response = await self._send_handling_auth(
File "httpx/_client.py", line 1689, in _send_handling_auth
response = await self._send_handling_redirects(
File "httpx/_client.py", line 1726, in _send_handling_redirects
response = await self._send_single_request(request)
File "httpx/_client.py", line 1763, in _send_single_request
response = await transport.handle_async_request(request)
File "app/http/pool.py", line 16, in handle_async_request
for attempt in stamina.retry_context(
File "stamina/_core.py", line 439, in __iter__
for r in _t.Retrying(
File "__init__.py", line 347, in __iter__
do = self.iter(retry_state=retry_state)
File "__init__.py", line 325, in iter
raise retry_exc.reraise()
File "__init__.py", line 158, in reraise
raise self.last_attempt.result()
File "concurrent/futures/_base.py", line 451, in result
return self.__get_result()
File "concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "app/http/pool.py", line 26, in handle_async_request
return await self._wrapper.handle_async_request(request)
File "httpx/_transports/default.py", line 372, in handle_async_request
with map_httpcore_exceptions():
File "contextlib.py", line 153, in __exit__
self.gen.throw(typ, value, traceback)
File "httpx/_transports/default.py", line 86, in map_httpcore_exceptions
raise mapped_exc(message) from exc
Is there any direction you could point me to debug this issue, or is it a bug in the library?
Any help is appreciated.
Originally, I wanted to get this into 23.1.0, but between work, having a life, and EP preparation I had make a decision.
I think it should be possible with contextvars, but it's a bit more involved, than one would expect.
This is a test that needs to pass:
def test_retry_block_no_recursive_by_default():
"""
When retrying context managers, don't retry recursively by default.
"""
inner = outer = 0
with pytest.raises(ValueError):
for o_a in stamina.retry_context(
on=ValueError, attempts=2, wait_max=0
):
with o_a:
outer += 1
for i_a in stamina.retry_context(
on=ValueError, attempts=2, wait_max=0
):
with i_a:
inner += 1
raise ValueError
assert 2 == inner
assert 2 == outer
Start up time which is important for CLIs:
❯ python -m timeit -n 1 -r 1 "import structlog"
1 loop, best of 1: 185 msec per loop
❯ python -m timeit -n 1 -r 1 "from prometheus_client import Counter"
1 loop, best of 1: 97.2 msec per loop
RSS which is important for long-running processes on users' machines:
❯ python -c "import psutil;f=lambda p=psutil.Process(): print(p.memory_info().rss / 1024 ** 2);f();import structlog;f()"
14.984375
29.125
❯ python -c "import psutil;f=lambda p=psutil.Process(): print(p.memory_info().rss / 1024 ** 2);f();from prometheus_client import Counter;f()"
14.6796875
22.03515625
In [11]: stamina.set_active(False)
...: for attempt in stamina.retry_context(on=ValueError, wait_initial=1, wait_max=2, timeout=40, wait_jitter=0, wait_exp_base=2, attempts=0):
...: with attempt:
...: print(attempt)
...: time.sleep(1)
...: print("test")
<Attempt num=1>
test
<Attempt num=1>
test
When set active is set to True it behaves like it should.
In [11]: stamina.set_active(True)
...: for attempt in stamina.retry_context(on=ValueError, wait_initial=1, wait_max=2, timeout=40, wait_jitter=0, wait_exp_base=2, attempts=0):
...: with attempt:
...: print(attempt)
...: time.sleep(1)
...: print("test")
<Attempt num=1>
test
Hi there,
I use stamina
to retry my async function with httpx.AsyncClient
, but once the request received httpx.RemoteProtocolError
, the program died without retry.
Here is how I use stamina
:
@stamina.retry(on=Exception, attempts=3, timeout=30)
async def func(url: str, params: dict, client: httpx.AsyncClient):
response: httpx.Response = await client.post(url, json=params)
response.raise_for_status()
Sorry I can not give you a snippet code to reproduce the problem because it just happens almost in 1/100k. I think the problem is that the server drop the connection and httpx still use that connection to send request. If this is the situation, maybe retry will not solve the problem either.
stamina version: 23.3.0
httpx version: 0.26.0
Below is the traceback:
Traceback (most recent call last):
File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpx/_transports/default.py", line 67, in map_httpcore_exceptions
yield
File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpx/_transports/default.py", line 371, in handle_async_request
resp = await self._pool.handle_async_request(req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 268, in handle_async_request
raise exc
File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 251, in handle_async_request
response = await connection.handle_async_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpcore/_async/connection.py", line 103, in handle_async_request
return await self._connection.handle_async_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpcore/_async/http11.py", line 133, in handle_async_request
raise exc
File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpcore/_async/http11.py", line 111, in handle_async_request
) = await self._receive_response_headers(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpcore/_async/http11.py", line 176, in _receive_response_headers
event = await self._receive_event(timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpcore/_async/http11.py", line 226, in _receive_event
raise RemoteProtocolError(msg)
httpcore.RemoteProtocolError: Server disconnected without sending a response.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/cms-test/text/netease.py", line 187, in query_text_async_helper
response: httpx.Response = await client.post(
^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpx/_client.py", line 1877, in post
return await self.request(
^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpx/_client.py", line 1559, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpx/_client.py", line 1646, in send
response = await self._send_handling_auth(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpx/_client.py", line 1674, in _send_handling_auth
response = await self._send_handling_redirects(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpx/_client.py", line 1711, in _send_handling_redirects
response = await self._send_single_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpx/_client.py", line 1748, in _send_single_request
response = await transport.handle_async_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpx/_transports/default.py", line 370, in handle_async_request
with map_httpcore_exceptions():
File "/opt/conda/envs/cms-test/lib/python3.12/contextlib.py", line 155, in __exit__
self.gen.throw(value)
File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpx/_transports/default.py", line 84, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.RemoteProtocolError: Server disconnected without sending a response.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.