gfmio / asyncio-gevent Goto Github PK

View Code? Open in Web Editor NEW

64.0 64.0 14.0 52 KB

asyncio & gevent in harmony

License: MIT License

Dockerfile 5.33% Python 94.67%

asyncio-gevent's Introduction

Hi there 👋

asyncio-gevent's People

Contributors

Stargazers

Watchers

Forkers

volttron bonicim steverice heuripedes bidaya0 richgong vraja2 skykistler thatguyjustin macro1 apploitech idealprediction

asyncio-gevent's Issues

When using multiple event loops, asyncio will raise an error.

When running the following program, you will get a 'RuntimeError: Cannot run the event loop while another loop is running' error. If the patch is not applied, the program will run normally.

import asyncio
import threading
import time
import gevent
import gevent.monkey
gevent.monkey.patch_all()
import asyncio_gevent

asyncio.set_event_loop_policy(asyncio_gevent.EventLoopPolicy())

def _start_event_loop(loop) -> None:
asyncio.set_event_loop(loop)
loop.run_forever()
loop.close()

def _get_threaded_loop():
new_loop = asyncio.new_event_loop()
thread_loop = threading.Thread(target=_start_event_loop, args=(new_loop,), daemon=True)
thread_loop.start()
return new_loop

async def test_async():
while True:
await asyncio.sleep(3)
print("test", time.time())

async def test_async2():
while True:
await asyncio.sleep(3)
print("test2", time.time())

if name == "main":
_loop = _get_threaded_loop()
asyncio.run_coroutine_threadsafe(test_async(), _loop)
asyncio.run(test_async2())

100% CPU utilisation

When launching asyncio.run_forever using event.spawn, the CPU utilisation goes to 100%.

The root cause is that the selector currently does not block by default but rather has a 0 timeout.

Is this production ready?

Basically, are there any know issues or limitations?

I saw that on the README it says running gevent on asyncio (by using asyncio as gevent's event loop, still work in progress). What does that WIP mens for that case in specific?

Cannot run asyncio.sleep in a greenlet

I want to run an asyncio-based function inside a greenlet but I am running into an issue where the eventloop is started whilst another is already running. Here's a minimal example:

import gevent.monkey
gevent.monkey.patch_all()
import asyncio
import asyncio_gevent

asyncio.set_event_loop_policy(asyncio_gevent.EventLoopPolicy())

import gevent

@asyncio_gevent.async_to_sync
async def async_main():
    await asyncio.sleep(1)


def main():
    gevent.joinall([
        gevent.spawn(async_main),
        gevent.spawn(async_main),
    ])

if __name__ == "__main__":
    main()

This will fail with the following output:

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 596, in run_forever
    self._check_running()
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 590, in _check_running
    raise RuntimeError(
RuntimeError: Cannot run the event loop while another loop is running
2024-01-26T10:07:42Z <Greenlet at 0x7fdd85be36a0: <bound method BaseEventLoop.run_forever of <EventLoop running=False closed=False debug=False>>> failed with RuntimeError

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 651, in run_until_complete
    raise RuntimeError('Event loop stopped before Future completed.')
RuntimeError: Event loop stopped before Future completed.
2024-01-26T10:07:42Z <Greenlet at 0x7fdd85be3560: <bound method BaseEventLoop.run_until_complete of <EventLoop running=False closed=False debug=False>>(<Task pending name='Task-2' coro=<async_main() run)> failed with RuntimeError

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run
  File "/usr/local/lib/python3.11/site-packages/asyncio_gevent/future_to_greenlet.py", line 118, in _run
    return ensured_future.result()
           ^^^^^^^^^^^^^^^^^^^^^^^
asyncio.exceptions.InvalidStateError: Result is not set.
2024-01-26T10:07:42Z <Greenlet at 0x7fdd85be2de0: _run(future=<coroutine object async_main at 0x7fdd85d82980>, loop=None, autostart_future=True, on_cancelled=<function future_to_greenlet.<locals>.on_cancelled)> failed with InvalidStateError

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run
  File "/usr/local/lib/python3.11/site-packages/asyncio_gevent/async_to_sync.py", line 37, in fn
    return greenlet.get()
           ^^^^^^^^^^^^^^
  File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get
  File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception
  File "/usr/local/lib/python3.11/site-packages/gevent/_compat.py", line 66, in reraise
    raise value.with_traceback(tb)
  File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run
  File "/usr/local/lib/python3.11/site-packages/asyncio_gevent/future_to_greenlet.py", line 118, in _run
    return ensured_future.result()
    ^^^^^^^^^^^^^^^^^
asyncio.exceptions.InvalidStateError: Result is not set.
2024-01-26T10:07:42Z <Greenlet at 0x7fdd85be2ac0: fn> failed with InvalidStateError

sys:1: RuntimeWarning: coroutine 'async_main' was never awaited
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
Task was destroyed but it is pending!
task: <Task pending name='Task-2' coro=<async_main() running at /code/src/comsdk/cli/main.py:10>>

I suspect that I am using either asyncio, gevent or asyncio-gevent wrongly here.

python 3.11 build

Hi folks,
Nice project.
Just tried building with py 3.11.0 and I think its giving a similar err to gevents build a few weeks back, It failed with error: unknown type name ‘CFrame’.

Cheers

Has anyone used this with websockets

I've been trying to get this to work with locust (based on greenlets) and the websockets library (based on asyncio)

But just get various errors related to 'tasks being on the wrong event loop' or an 'event loop already running' has anyone done something similar to this?

A more detailed issue has been raised here on the locust repo: locustio/locust#2383

example for asyncio on gevent does not work

I tried to run the example to run "asyncio on gevent" but it does not work for me:

import gevent.monkey
gevent.monkey.patch_all()
import asyncio
import asyncio_gevent
# example in README.md uses:
#   asyncio.set_default_event_loop_policy(asyncio_gevent.EventLoopPolicy)
# however this function does not exist in Python 3.9
asyncio.set_event_loop_policy(asyncio_gevent.EventLoopPolicy())

async def async_function() -> int:
    await asyncio.sleep(1)
    return 42

def main() -> None:
    future = async_function()
    # blocking in ".yield_future()"
    greenlet = asyncio_gevent.yield_future(future)
    greenlet.join()

main()

Initially I noticed that asyncio.set_default_event_loop_policy() does not exist for me (Fedora 34, Python 3.9). I used .set_event_loop_policy() but then execution blocks in yield_future().

I guess got the initialization wrong somehow but I can't figure out what is wrong here...

ctrl-c / KeyboardInterrupt makes event loop exceptions

Exiting asyncio.run(<a main function()>) with SIGINT (ctrl-c, KeyboardInterrupt exception) makes bad RuntimeError exceptions. I am affected by this in a prompt-toolkit application relying on gevent.

The minimum code to reproduce the problem:

import gevent
import gevent.monkey; gevent.monkey.patch_all()
import asyncio
import asyncio_gevent
import time

asyncio.set_event_loop_policy(asyncio_gevent.EventLoopPolicy())

async def my_main():
    gevent.sleep(10)

# press CTRL-C to exit the Python program before 10 seconds
asyncio.run(my_main())

python test.py
^CKeyboardInterrupt
2022-11-16T14:00:43Z
Traceback (most recent call last):
  File "/home/matias/miniconda3/envs/bliss/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/home/matias/miniconda3/envs/bliss/lib/python3.7/asyncio/base_events.py", line 574, in run_until_complete
    self.run_forever()
  File "/home/matias/dev/asyncio-gevent/asyncio_gevent/event_loop.py", line 19, in run_forever
    greenlet.join()
  File "src/gevent/greenlet.py", line 833, in gevent._gevent_cgreenlet.Greenlet.join
  File "src/gevent/greenlet.py", line 859, in gevent._gevent_cgreenlet.Greenlet.join
  File "src/gevent/greenlet.py", line 848, in gevent._gevent_cgreenlet.Greenlet.join
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/matias/miniconda3/envs/bliss/lib/python3.7/asyncio/runners.py", line 46, in run
    _cancel_all_tasks(loop)
  File "/home/matias/miniconda3/envs/bliss/lib/python3.7/asyncio/runners.py", line 62, in _cancel_all_tasks
    tasks.gather(*to_cancel, loop=loop, return_exceptions=True))
  File "/home/matias/miniconda3/envs/bliss/lib/python3.7/asyncio/base_events.py", line 563, in run_until_complete
    self._check_runnung()
  File "/home/matias/miniconda3/envs/bliss/lib/python3.7/asyncio/base_events.py", line 523, in _check_runnung
    raise RuntimeError('This event loop is already running')
RuntimeError: This event loop is already running

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/bla.py", line 12, in <module>
    asyncio.run(my_main())
  File "/home/matias/miniconda3/envs/bliss/lib/python3.7/asyncio/runners.py", line 50, in run
    loop.close()
  File "/home/matias/miniconda3/envs/bliss/lib/python3.7/asyncio/unix_events.py", line 55, in close
    super().close()
  File "/home/matias/miniconda3/envs/bliss/lib/python3.7/asyncio/selector_events.py", line 88, in close
    raise RuntimeError("Cannot close a running event loop")
RuntimeError: Cannot close a running event loop

How to use loop.run_forever?

Hi, this project is great! Thanks for putting it together.

I've been experimenting with running asyncio functions in gevent with loop.run_forever. I had trouble getting the asyncio loop to actually run any scheduled tasks because the event loop would never be notified when create_task is called because calling create_task does not notify selector.select. I was able to resolve this by making EventLoop.call_soon call self._write_to_self() in the same way that EventLoop.call_soon_threadsafe does, to notify the selector.

Something like this:

class GeventEventloop(asyncio.SelectorEventLoop):
    def __init__(self, selector=None):
        super(GeventEventloop, self).__init__(selector or gevent.selectors.DefaultSelector())

    def call_soon(self, callback, *args, context=None):
        handle = super(GeventEventloop, self).call_soon(callback, *args, context=context)
        self._write_to_self() # Without this, it never runs the task function
        return handle


loop = GeventEventloop()
loop_greenlet = gevent.spawn(loop.run_forever)
loop_greenlet.start()


async def logit(text):
    print(text)

def callit(fn, *args):
    task = loop.create_task(fn(*args))
    event = gevent.event.Event()
    def done(_):
        event.set()
    task.add_done_callback(done)
    event.wait()
    return task.result()

greenlets = []
for i in range(20):
    greenlets.append(gevent.spawn(callit, logit, "meow"))

gevent.wait(greenlets)

I'm curious if I'm doing something wrong, or if there is something useful to add to the framework or docs.

Is this module the magic that I need?

I have described my problem in https://stackoverflow.com/questions/76880897/django-running-async-code-in-wsgi-mode-causes-synchronousonlyoperation-error . Basically, I am running a Django application on top of unicorn --worker-class gevent. I implemented one view function using asyncio and received SynchronousOnlyOperation exceptions.

I do not quite understand how gevent, asyncio, gunicorn interacts, but is this module the one that I need to resolve the problem? The documentation seems to suggest that I only need to use gevent's event loop for asyncio, but I am not quite sure if it will work.

Killing greenlet returned from future_to_greenlet throws AttributeError: 'coroutine' object has no attribute 'cancel'

I'm chasing down a 100% CPU consumption problem that is somewhere in gevent's hub run while running greenlets from future_to_greenlet.

While investigating, I wanted to make sure that greenlets were killed after being joined. (For context, I'm creating and running these futures in parallel using locust so it's a bit tricky to make sure everything is getting cleaned up.)

My code looked something like this:

future = my_async_function()
greenlet = asyncio_gevent.future_to_greenlet(future)
greenlet.start()
greenlet.join(timeout=timeout_seconds)
greenlet.kill()

But I was receiving an AttributeError: 'coroutine' object has no attribute 'cancel' during greenlet.kill(), thrown from https://github.com/gfmio/asyncio-gevent/blob/main/asyncio_gevent/future_to_greenlet.py#L61

Should this instead be future.close()? Coroutine documentation is here. This seems to work for me. My example now looks like:

future = my_async_function()
greenlet = asyncio_gevent.future_to_greenlet(future, autocancel_future=False)
greenlet.start()
greenlet.join(timeout=timeout_seconds)
greenlet.kill()
future.close()