python-trio / async_generator Goto Github PK
View Code? Open in Web Editor NEWMaking it easy to write async iterators in Python 3.5
License: Other
Making it easy to write async iterators in Python 3.5
License: Other
I was expecting something like the following example to work, but it does not. This originally came up in: fastapi/fastapi#1204
from async_generator import asynccontextmanager
class GenClass:
async def __call__(self):
yield "hello"
cm = asynccontextmanager(GenClass())
with cm as value:
print(value)
The result is an exception:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/.../lib/python3.6/site-packages/async_generator/_util.py", line 100, in asynccontextmanager
"must be an async generator (native or from async_generator; "
TypeError: must be an async generator (native or from async_generator; if using @async_generator then @acontextmanager must be on top.
I saw your pycon lightning talk on youtube and thought it was pretty neat. await yield
for async for
has been something I've wanted since I first read about async for
. I have made decorators that try to take care of all the __aiter__
/__anext__
boilerplate code in the past, but I have never seen one that used a yield_
function like yours (I have always used queues to pass values). It's a pretty clever idea.
As I was reading through your code and testing differences between your implementation and mine, I had a few questions:
YieldWrapper
?It looks like you are using it as a sentinal value to determine when you've reached the end of the coroutine chain, but why not just yield the value directly and then in send() you can use inspect.isawaitable(result)
to determine if you're at the end. I don't know if you would need inspect.isgenerator
or inspect.iscoroutine
, but for all the tests I've done, inspect.isawaitable(result)
has been sufficient.
This would have the added benefit of allowing the decorator to turn any generator into an async iterable (not just ones using yield_
).
__anext__
call?It seems like unnecessary overhead to create a new instance for each __anext__
call, why not just create an instance in __aiter__
(which is called once per async for
) and give it a __anext__
that returns self
? This also separates them more into an "async iterable" and "async iterator".
yield_from_
?I read your README, and I agree that an asend
or athrow
seem pretty weird, but I don't understand the problem with yield_from_
. The most obvious and useful case I can think of is generator delegation, and as far as I can tell your (commented out) version does that just fine.
For reference, here's the version I was using (updated to adopt your yield_
ideas and be more flexible): https://gist.github.com/bj0/0313ee67766de6a04881
I was asked to raise this here from Fuyukai/Kyoukai#19
I get a TypeError, bad argument type for built-in operation
within Kyoukai, at an import statement (within asyncio_extras
: from async_generator import yield_
.
Python version: Python 3.7.6+ (heads/3.7:b0a6ec256b)
OS: macOS 10.14.6
async_generator.asynccontextmanager
started to fail in python 3.7-dev
(which is also the 3.7-dev
in pyenv
).
If we change to use contextlib.asynccontextmanager
, it works. It's why I'm thinking something wrong in async_generator
when an async-generator is being closed. I'm sorry that I haven't dug into the cause.
This issue seems to happen since python/cpython@b76d5e9.
# example_code.py
import asyncio
from async_generator import asynccontextmanager
@asynccontextmanager
async def async_iterator():
yield 1
async def run():
async with async_iterator():
...
asyncio.run(run())
# Output of the executed code
an error occurred during closing of asynchronous generator <async_generator object async_iterator at 0x108632290>
asyncgen: <async_generator object async_iterator at 0x108632290>
Traceback (most recent call last):
File "/Users/mhchia/.pyenv/versions/3.7-dev/lib/python3.7/asyncio/runners.py", line 43, in run
return loop.run_until_complete(main)
File "/Users/mhchia/.pyenv/versions/3.7-dev/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
return future.result()
File "/Users/mhchia/projects/practice/python/async-generator/await-twice.py", line 13, in run
...
File "/Users/mhchia/.pyenv/versions/3.7-dev/envs/py-libp2p-3.7-dev/lib/python3.7/site-packages/async_generator/_util.py", line 84, in __aexit__
raise
File "/Users/mhchia/.pyenv/versions/3.7-dev/envs/py-libp2p-3.7-dev/lib/python3.7/site-packages/async_generator/_util.py", line 14, in __aexit__
await self._aiter.aclose()
RuntimeError: cannot reuse already awaited aclose()/athrow()
During handling of the above exception, another exception occurred:
RuntimeError: cannot reuse already awaited aclose()/athrow()
Traceback (most recent call last):
File "/Users/mhchia/projects/practice/python/async-generator/await-twice.py", line 16, in <module>
asyncio.run(run())
File "/Users/mhchia/.pyenv/versions/3.7-dev/lib/python3.7/asyncio/runners.py", line 43, in run
return loop.run_until_complete(main)
File "/Users/mhchia/.pyenv/versions/3.7-dev/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
return future.result()
File "/Users/mhchia/projects/practice/python/async-generator/await-twice.py", line 13, in run
...
File "/Users/mhchia/.pyenv/versions/3.7-dev/envs/py-libp2p-3.7-dev/lib/python3.7/site-packages/async_generator/_util.py", line 84, in __aexit__
raise
File "/Users/mhchia/.pyenv/versions/3.7-dev/envs/py-libp2p-3.7-dev/lib/python3.7/site-packages/async_generator/_util.py", line 14, in __aexit__
await self._aiter.aclose()
RuntimeError: cannot reuse already awaited aclose()/athrow()
should be de-deprecated ASAP
I have an abstract class that defines an abstract async context manager (Python 3.7 code):
import abc
import contextlib
class Foo(metaclass=abc.ABCMeta):
@abc.abstractmethod
@contextlib.asynccontextmanager
async def bar(self):
pass
In order to make it compatible with Python 3.6, I would use async_generator.asynccontextmanager
import async_generator
class Foo(metaclass=abc.ABCMeta):
@abc.abstractmethod
@async_generator.asynccontextmanager
async def bar(self):
pass
However, this raises an error:
TypeError: must be an async generator (native or from async_generator; if using @async_generator then @acontextmanager must be on top.
This can be fixed using async_generator.async_generator
, but I think that was intended for Python3.5 (which lacked native async generators) and not for Python3.6.
class Foo(metaclass=abc.ABCMeta):
@abc.abstractmethod
@async_generator.asynccontextmanager
@async_generator.async_generator
async def bar(self):
pass
This seems to work on Python3.5 too.
I can extend the class and implement the method without using async_generator.async_generator
in both, Python3.6 and Python3.7 (as expected):
class Bar(Foo):
@async_generator.asynccontextmanager
async def bar(self):
print('Before')
yield 42
print('After')
bar = Bar()
async with bar.bar() as context:
print('Context:', context)
Maybe the docs / readme could include an abstract method example.
Or maybe Python3.6 should work without adding @async_generator
.
I don't know if this is intentional, but contextlib's asynccontextmanager
decorated functions return None whereas here they'd return False. The snippet below shows that:
import asyncio
import contextlib
from async_generator import asynccontextmanager
@contextlib.asynccontextmanager
async def f():
yield
@asynccontextmanager
async def g():
yield
async def main():
ctx1 = f()
ctx2 = g()
await asyncio.gather(*(ctx.__aenter__() for ctx in [ctx1, ctx2]))
exits = await asyncio.gather(*(ctx.__aexit__(None, None, None) for ctx in [ctx1, ctx2]))
print(exits)
l = asyncio.get_event_loop()
l.run_until_complete(main())
async_generator is really useful! But I see one surprising behavior. When I run this code:
import trio
from async_generator import async_generator, yield_
@async_generator
async def items():
for i in range(10):
await yield_(i)
async def main():
async for i in items():
if i > 4:
break
print(i)
if __name__ == '__main__':
trio.run(main)
I get the following exception:
Exception ignored in: <bound method AsyncGenerator.__del__ of <async_generator._impl.AsyncGenerator object at 0x110a9b908>>
Traceback (most recent call last):
File ".../python3.5/site-packages/async_generator/_impl.py", line 324, in __del__
.format(self._coroutine.cr_frame.f_code.co_name)
RuntimeError: partially-exhausted async_generator 'items' garbage collected
Is this expected? The same code using the Python 3.6 syntax does not print any exception. Thanks!
should probably figure out why and fix it at some point
I'm trying to package your module as an rpm package. So I'm using the typical build, install and test cycle used on building packages from non-root account.
May I ask for help because few units are failing:
+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-async-generator-1.10-12.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-async-generator-1.10-12.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.12, pytest-6.2.5, py-1.10.0, pluggy-0.13.1
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
Using --randomly-seed=2477080430
rootdir: /home/tkloczko/rpmbuild/BUILD/async_generator-1.10
plugins: forked-1.3.0, shutil-1.7.0, virtualenv-1.7.0, expect-1.1.0, flake8-1.0.7, timeout-1.4.2, betamax-0.8.1, freezegun-0.4.2, aspectlib-1.5.2, toolbox-0.5, rerunfailures-9.1.1, requests-mock-1.9.3, cov-2.12.1, flaky-3.7.0, benchmark-3.4.1, xdist-2.3.0, pylama-7.7.1, datadir-1.3.1, regressions-2.2.0, cases-3.6.3, xprocess-0.18.1, black-0.3.12, anyio-3.3.0, asyncio-0.15.1, trio-0.7.0, subtests-0.5.0, isort-2.0.0, hypothesis-6.14.6, mock-3.6.1, profiling-1.7.0, randomly-3.8.0, Faker-8.12.1, nose2pytest-1.0.8, pyfakefs-4.5.1, tornado-0.8.1, twisted-1.13.3
collected 0 items / 1 error
================================================================================== ERRORS ==================================================================================
______________________________________________________________________ ERROR collecting test session _______________________________________________________________________
/usr/lib/python3.8/site-packages/_pytest/config/__init__.py:570: in _importconftest
mod = import_path(conftestpath, mode=importmode)
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:544: in import_path
raise ImportPathMismatchError(module_name, module_file, path)
E _pytest.pathlib.ImportPathMismatchError: ('async_generator._tests.conftest', '/home/tkloczko/rpmbuild/BUILDROOT/python-async-generator-1.10-12.fc35.x86_64/usr/lib/python3.8/site-packages/async_generator/_tests/conftest.py', PosixPath('/home/tkloczko/rpmbuild/BUILD/async_generator-1.10/async_generator/_tests/conftest.py'))
========================================================================= short test summary info ==========================================================================
ERROR - _pytest.pathlib.ImportPathMismatchError: ('async_generator._tests.conftest', '/home/tkloczko/rpmbuild/BUILDROOT/python-async-generator-1.10-12.fc35.x86_64/usr/l...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================================= 1 error in 0.40s =============================================================================
pytest-xprocess reminder::Be sure to terminate the started process by running 'pytest --xkill' if you have not explicitly done so in your fixture with 'xprocess.getinfo(<process_name>).terminate()'.
and the same but with --import-mode=importlib
+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-async-generator-1.10-12.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-async-generator-1.10-12.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra --import-mode=importlib
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.12, pytest-6.2.5, py-1.10.0, pluggy-0.13.1
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
Using --randomly-seed=1808263463
rootdir: /home/tkloczko/rpmbuild/BUILD/async_generator-1.10
plugins: forked-1.3.0, shutil-1.7.0, virtualenv-1.7.0, expect-1.1.0, flake8-1.0.7, timeout-1.4.2, betamax-0.8.1, freezegun-0.4.2, aspectlib-1.5.2, toolbox-0.5, rerunfailures-9.1.1, requests-mock-1.9.3, cov-2.12.1, flaky-3.7.0, benchmark-3.4.1, xdist-2.3.0, pylama-7.7.1, datadir-1.3.1, regressions-2.2.0, cases-3.6.3, xprocess-0.18.1, black-0.3.12, anyio-3.3.0, asyncio-0.15.1, trio-0.7.0, subtests-0.5.0, isort-2.0.0, hypothesis-6.14.6, mock-3.6.1, profiling-1.7.0, randomly-3.8.0, Faker-8.12.1, nose2pytest-1.0.8, pyfakefs-4.5.1, tornado-0.8.1, twisted-1.13.3
collected 0 items / 2 errors
================================================================================== ERRORS ==================================================================================
_____________________________________________________ ERROR collecting async_generator/_tests/test_async_generator.py ______________________________________________________
ImportError while importing test module '/home/tkloczko/rpmbuild/BUILD/async_generator-1.10/async_generator/_tests/test_async_generator.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
async_generator/_tests/test_async_generator.py:9: in <module>
from .conftest import mock_sleep
E ImportError: attempted relative import with no known parent package
___________________________________________________________ ERROR collecting async_generator/_tests/test_util.py ___________________________________________________________
ImportError while importing test module '/home/tkloczko/rpmbuild/BUILD/async_generator-1.10/async_generator/_tests/test_util.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
async_generator/_tests/test_util.py:3: in <module>
from .. import aclosing, async_generator, yield_, asynccontextmanager
E ImportError: attempted relative import with no known parent package
========================================================================= short test summary info ==========================================================================
ERROR async_generator/_tests/test_async_generator.py
ERROR async_generator/_tests/test_util.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================================ 2 errors in 0.40s =============================================================================
pytest-xprocess reminder::Be sure to terminate the started process by running 'pytest --xkill' if you have not explicitly done so in your fixture with 'xprocess.getinfo(<process_name>).terminate()'.
Hi,
On pypi the 1.10 is already published. it would be nice to have a corresponding git commit available via a pushed git tag.
Can you please tag and push 1.10 to github?
cheers
Now that Python 3.6 reached EOL, I believe this library does more harm than good and we should deprecate it and eventually archive it. What do you think?
Is it possible to create a new release/tag? The latest one if from January and there have been non-insignificant fixes since.
Thank you.
In private email, Nicolas Boulenguez reported the following issues they encountered while working on packaging async_generator for Debian:
The 1.9 tag is missing from Github.
The source says that the license is "MIT or APACHE2 (your choice)" but the 1.9 tarball on pypi only mentions MIT.
The 1.9 tarball on pypi contains no manual, neither the source nor the formatted version.
If this is easy for you to do, please consider removing stuff specific to github like .gitignore from the release tarball. People like me downloading the tarball instead of cloning the repository might have a different workflow with their own .gitignore settings. If removing files from the tarball is convoluted with github, forget this suggestion. This is only a matter of convenience.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.