magicstack / uvloop Goto Github PK
View Code? Open in Web Editor NEWUltra fast asyncio event loop.
License: Apache License 2.0
Ultra fast asyncio event loop.
License: Apache License 2.0
Creating a uvloop.Loop
has the surprising side effect of marking file descriptors it doesn't own (notably including stdin/out/err) as FD_CLOEXEC
. This causes any use of subprocesses which do not explicitly assign stdin/out/err to fail, sometimes in surprising ways. This toy example should print "hi", but does nothing if a uvloop.Loop
has been created. In more complex examples a subprocess may open other files or sockets, those sockets get assigned the first three file descriptors, and its stdio output gets interleaved with its network traffic.
import uvloop
import subprocess
uvloop.Loop()
p = subprocess.Popen(["echo", "hi"])
p.wait()
Requires brew install libtool
because of autogen.sh: line 43: glibtoolize: command not found
Hello,
I thought I'd give the aiofiles tests a go on uvloop and had a few failing. Here's a minimal program that works under the default asyncio, and doesn't with uvloop:
import asyncio
import socket
from os.path import join, dirname
import uvloop
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
async def test_sendfile_socket():
"""Test the sendfile functionality, file-to-socket."""
port = 50000
@asyncio.coroutine
def serve_file(_, writer):
pass
server = await asyncio.start_server(serve_file, port=port)
await server.wait_closed()
loop = asyncio.get_event_loop()
loop.run_until_complete(test_sendfile_socket())
> python t.py
Traceback (most recent call last):
File "uvloop/loop.pyx", line 1168, in uvloop.loop.Loop.create_server (uvloop/loop.c:19802)
File "uvloop/handles/streamserver.pyx", line 38, in uvloop.loop.UVStreamServer.listen (uvloop/loop.c:56668)
File "uvloop/handles/streamserver.pyx", line 75, in uvloop.loop.UVStreamServer._fatal_error (uvloop/loop.c:57149)
OSError: [Errno 98] Address already in use
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "t.py", line 25, in <module>
loop.run_until_complete(test_sendfile_socket())
File "uvloop/loop.pyx", line 1015, in uvloop.loop.Loop.run_until_complete (uvloop/loop.c:18124)
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 240, in _step
result = coro.send(None)
File "t.py", line 18, in test_sendfile_socket
server = await asyncio.start_server(serve_file, port=port)
File "/usr/lib/python3.5/asyncio/streams.py", line 116, in start_server
return (yield from loop.create_server(factory, host, port, **kwds))
File "uvloop/loop.pyx", line 1172, in create_server (uvloop/loop.c:19927)
OSError: [Errno 98] error while attempting to bind on address ('::', 50000, 0, 0): address already in use
64-bit Ubuntu 16.04, Python 3.5. Using reuse_port=True
will solve it, not sure if that's the difference.
loop.create_connection
, loop.create_unix_connection
, loop.create_server
, loop.create_unix_server
I've read the libuv documentation and it appears that it is, and that uvloop is calling directly into it. So if I use aiohttp to do requests will the dns resolution be async as well?
And do you know why node still uses c-ares to do dns resolution if libuv is fully async?
Hi,
I tried to build the project to run the test on my machine (Ubuntu 14.04, 64b).
First I had to install automake and libtool (you may want to add this step on the readme), then running make I got:
building 'uvloop.loop' extension
creating build
creating build/temp.linux-x86_64-3.5
creating build/temp.linux-x86_64-3.5/uvloop
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.5m -I/home/user/.local/share/virtualenvs/ace813a9968fde5/include/python3.5m -Ivendor/libuv/include -c uvloop/loop.c -o build/temp.linux-x86_64-3.5/uvloop/loop.o
x86_64-linux-gnu-gcc: error: uvloop/loop.c: No such file or directory
x86_64-linux-gnu-gcc: fatal error: no input files
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 4
make: *** [compile] Erreur 1
I need some suggestions for debugging server startup hangs.
Startup code looks like:
cr = loop.create_server(self.factory, addr[0], addr[1],
reuse_address=True, ssl=ssl)
f = asyncio.async(cr, loop=loop)
server = loop.run_until_complete(f)
What's sometimes happening, in the context of the ZEO tests, is that the run_loop_until_complete
call above never returns.
This is in a port of ZEO, github.com/zopefoundation/ZEO, to asyncio. The ZEO tests are exhaustive, and I suspect in uvloop's case, exhausting. :) The tests start and stop servers 10s, maybe 100s of times in a test runs that typically lasts a few minutes. If I run a smaller subset of the tests I don't get a hang.
Most tests run the server as a subprocess using multiprocessing. Some tests run the server using threading. It appears that the server isn't exiting, but merely hanging.
Do you have any suggestions for how to debug this? I've tried setting PYTHONASYNCIODEBUG=1 and enabling debug logging, but that isn't yielding any additional information..
Build is successful on Solaris 11, but the uvloop.so file is missing references to kstat_* methods. To correct the issue the the gcc linker needs -lkstat
The below patch fixes the issue.
diff --git a/setup.py b/setup.py
index ba7bff2..e30783f 100644
--- a/setup.py
+++ b/setup.py
@@ -86,6 +86,8 @@ class libuv_build_ext(build_ext):
self.compiler.add_library('rt')
elif sys.platform.startswith('freebsd'):
self.compiler.add_library('kvm')
+ elif sys.platform.startswith('sunos'):
+ self.compiler.add_library('kstat')
super().build_extensions()
If there are other C libraries using libuv and expose python interface, is there a way to use the same uv event loop for uvloop and the library?
import asyncio
import os
if os.getenv('USE_UVLOOP'):
import uvloop
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
HOST="10.99.99.99" # should not exist on your network!
PORT=9090
async def doconnect(loop):
try:
return await asyncio.wait_for(asyncio.open_connection(HOST, PORT), timeout=2)
except asyncio.TimeoutError:
pass
loop=asyncio.get_event_loop()
loop.run_until_complete( doconnect(loop))
In classic asyncio eventloop this program has no output. With USE_UVLOOP=1 it logs the following at the ERROR level
Fatal error on transport TCPTransport (connect failed)
protocol: <asyncio.streams.StreamReaderProtocol object at 0x7fb567fa29e8>
transport: <TCPTransport closed=True reading=False 0xe823e8>
concurrent.futures._base.CancelledError
IMHO this should not be logged at this level.
Cancellation should be treated the same as the other exception types that are ignored in BaseTransport._fatal_error (around line 44)
I'm trying to use uvloop in my proxy project. The diff is below (or see it online here):
diff -r 4c40523b6830 -r 575a8ff4284f wormhole/proxy.py
--- a/wormhole/proxy.py Fri May 06 14:41:45 2016 +0700
+++ b/wormhole/proxy.py Tue May 10 09:44:00 2016 +0700
@@ -62,6 +62,12 @@
if not (1 <= args.port <= 65535):
parser.error('port must be 1-65535')
+ try:
+ import uvloop
+ except ImportError:
+ pass
+ else:
+ asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
loop = asyncio.get_event_loop()
try:
loop.run_until_complete(
It work pretty fast for a while, then everything just stop. I stop process by control-c. ps
show no process running. However, netstat -an
show a lot of CLOSE_WAIT
connections, and then I have to reboot my Mac to clear them.
Do I did something wrong in my code? It never happened when I use pure asyncio.
Trying out uvloop on the b2 build of python 3.6 fails to import..
Python 3.6.0b2 (default, Oct 12 2016, 20:13:22)
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import uvloop
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/mayfield/project/uvloop/uvloop/__init__.py", line 7, in <module>
from .loop import Loop as __BaseLoop, Future
File "uvloop/future.pyx", line 249, in init uvloop.loop (uvloop/loop.c:114059)
class Future(BaseFuture, aio_Future):
TypeError: multiple bases have instance lay-out conflict
When running pulsar in multiprocessing mode I get an error when using uvloop
as the event loop.
To reproduce the error try the same code in issue #54 with pulsar from master branch:
pip install git+http://github.com/quantmind/pulsar.git@master#egg=pulsar uvloop
py scripy.py --io uv --concurrency multi -w 2
uvloop complains about the socket being not valid (closed probably).
11:24:10 [p=13310, t=140736205472704, ERROR, asyncio] Task exception was never retrieved
future: <Task finished coro=<SocketServer.worker_start() done, defined at /Users/lsbardel/.pyenv/versions/3.5.2/lib/python3.5/site-packages/pulsar/apps/socket/__init__.py:270> exception=OSError(38, 'Socket operation on non-socket')>
Traceback (most recent call last):
File "uvloop/future.pyx", line 372, in uvloop.loop.BaseTask._fast_step (uvloop/loop.c:105885)
File "/Users/lsbardel/.pyenv/versions/3.5.2/lib/python3.5/site-packages/pulsar/apps/socket/__init__.py", line 274, in worker_start
server = await self.create_server(worker)
File "/Users/lsbardel/.pyenv/versions/3.5.2/lib/python3.5/site-packages/pulsar/apps/socket/__init__.py", line 323, in create_server
await server.start_serving(cfg.backlog, sslcontext=self.sslcontext())
File "/Users/lsbardel/.pyenv/versions/3.5.2/lib/python3.5/site-packages/pulsar/async/protocols.py", line 666, in start_serving
ssl=sslcontext)
File "uvloop/loop.pyx", line 1376, in create_server (uvloop/loop.c:26228)
File "uvloop/loop.pyx", line 1373, in uvloop.loop.Loop.create_server (uvloop/loop.c:26167)
File "uvloop/handles/streamserver.pyx", line 43, in uvloop.loop.UVStreamServer.listen (uvloop/loop.c:72688)
File "uvloop/handles/streamserver.pyx", line 80, in uvloop.loop.UVStreamServer._fatal_error (uvloop/loop.c:73173)
OSError: [Errno 38] Socket operation on non-socket
This occurs when using asyncio subprocess as process concurrency.
It looks like uvloop
has closed the socket in the main process.
Pulsar create the socket server in the main process, than loop.remove_reader
and pass the socket to subprocesses (implementation is https://github.com/quantmind/pulsar/blob/master/pulsar/apps/socket/__init__.py#L220 last five lines in the method).
This works with vanilla asyncio but it may not be the best way of doing it.
I notice that remove_reader
and remove_writer
are asynchronous with respect to when the file descriptor is actually removed from its epoll set (linux terminology). This is a byproduct of how libuv works.
This is different from the asyncio module and causes troubles with at least dup() based FDs. Namely serving static files with aiohttp under uvloop will induce a 100% CPU loop inside libuv because it can't remove the FD. Strace attached..
18:25:45.050832 open("/home/mayfield/project/aiocluster/aiocluster/diag/ui/node_modules/semantic-ui-css/semantic.css", O_RDONLY|O_CLOEXEC) = 24 <0.000031>
18:25:45.050911 ioctl(24, FIOCLEX) = 0 <0.000018>
18:25:45.050970 fstat(24, {st_mode=S_IFREG|0775, st_size=752358, ...}) = 0 <0.000019>
18:25:45.051036 ioctl(24, TCGETS, 0x7ffd90f38cb0) = -1 ENOTTY (Inappropriate ioctl for device) <0.000019>
18:25:45.051104 lseek(24, 0, SEEK_CUR) = 0 <0.000020>
18:25:45.051853 fcntl(23, F_DUPFD_CLOEXEC, 0) = 25 <0.000024>
18:25:45.051972 ioctl(25, FIONBIO, [0]) = 0 <0.000018>
18:25:45.052067 write(1, "\nCREATED", 8) = 8 <0.000039>
18:25:45.052225 getsockname(25, {sa_family=AF_INET, sin_port=htons(7878), sin_addr=inet_addr("192.168.17.147")}, [16]) = 0 <0.000023>
18:25:45.052336 getpeername(25, {sa_family=AF_INET, sin_port=htons(50857), sin_addr=inet_addr("192.168.17.179")}, [16]) = 0 <0.000022>
18:25:45.052443 write(1, " 25 <socket.socket fd=25, family"..., 159) = 159 <0.000026>
18:25:45.052521 ioctl(25, FIONBIO, [1]) = 0 <0.000017>
18:25:45.052603 sendto(25, "HTTP/1.1 200 OK\r\nContent-Type: t"..., 184, 0, NULL, 0) = 184 <0.000068>
18:25:45.052777 sendfile(25, 24, [0] => [143352], 752358) = 143352 <0.000083>
18:25:45.052970 getsockname(25, {sa_family=AF_INET, sin_port=htons(7878), sin_addr=inet_addr("192.168.17.147")}, [16]) = 0 <0.000019>
18:25:45.053048 getpeername(25, {sa_family=AF_INET, sin_port=htons(50857), sin_addr=inet_addr("192.168.17.179")}, [16]) = 0 <0.000017>
18:25:45.053131 write(1, "ADD 25 <socket.socket fd=25, fam"..., 144) = 144 <0.000024>
18:25:45.053228 epoll_ctl(6, EPOLL_CTL_ADD, 25, {EPOLLIN, {u32=4294967295, u64=18446744073709551615}}) = 0 <0.000021>
18:25:45.053289 epoll_ctl(6, EPOLL_CTL_DEL, 25, 0x7ffd90f38db0) = 0 <0.000048>
18:25:45.053384 ioctl(25, FIONBIO, [1]) = 0 <0.000021>
18:25:45.053499 epoll_ctl(6, EPOLL_CTL_ADD, 25, {EPOLLOUT, {u32=25, u64=25}}) = 0 <0.000022>
18:25:45.053584 epoll_wait(6, [], 1024, 0) = 0 <0.000019>
18:25:45.053651 epoll_wait(6, [{EPOLLOUT, {u32=25, u64=25}}], 1024, 43) = 1 <0.015515>
18:25:45.069554 getsockname(25, {sa_family=AF_INET, sin_port=htons(7878), sin_addr=inet_addr("192.168.17.147")}, [16]) = 0 <0.000181>
18:25:45.070023 getpeername(25, {sa_family=AF_INET, sin_port=htons(50857), sin_addr=inet_addr("192.168.17.179")}, [16]) = 0 <0.000158>
18:25:45.070464 write(1, "REMOVE! 25 <socket.socket fd=25,"..., 148) = 148 <0.000069>
18:25:45.070709 sendfile(25, 24, [143352] => [273672], 609006) = 130320 <0.000131>
18:25:45.071063 getsockname(25, {sa_family=AF_INET, sin_port=htons(7878), sin_addr=inet_addr("192.168.17.147")}, [16]) = 0 <0.000026>
18:25:45.071258 getpeername(25, {sa_family=AF_INET, sin_port=htons(50857), sin_addr=inet_addr("192.168.17.179")}, [16]) = 0 <0.000113>
18:25:45.071623 write(1, "ADD 25 <socket.socket fd=25, fam"..., 144) = 144 <0.000179>
18:25:45.071947 epoll_ctl(6, EPOLL_CTL_ADD, 25, {EPOLLIN, {u32=4294967295, u64=18446744073709551615}}) = -1 EEXIST (File exists) <0.000065>
18:25:45.072065 epoll_ctl(6, EPOLL_CTL_DEL, 25, 0x7ffd90f368d0) = 0 <0.000031>
18:25:45.072139 ioctl(25, FIONBIO, [1]) = 0 <0.000018>
18:25:45.072225 epoll_ctl(6, EPOLL_CTL_ADD, 25, {EPOLLOUT, {u32=25, u64=25}}) = 0 <0.000022>
18:25:45.072302 epoll_wait(6, [], 1024, 24) = 0 <0.024234>
18:25:45.097030 epoll_wait(6, [], 1024, 0) = 0 <0.000166>
18:25:45.097609 epoll_ctl(6, EPOLL_CTL_ADD, 15, {EPOLLIN, {u32=4294967295, u64=18446744073709551615}}) = -1 EEXIST (File exists) <0.000145>
18:25:45.098014 epoll_ctl(6, EPOLL_CTL_DEL, 15, 0x7ffd90f39590) = 0 <0.000049>
18:25:45.098193 ioctl(15, FIONBIO, [1]) = 0 <0.000030>
18:25:45.098361 epoll_ctl(6, EPOLL_CTL_ADD, 15, {EPOLLIN, {u32=15, u64=15}}) = 0 <0.000085>
18:25:45.098540 epoll_wait(6, [], 1024, 0) = 0 <0.000018>
18:25:45.098630 epoll_wait(6, [{EPOLLOUT, {u32=25, u64=25}}], 1024, 99) = 1 <0.004728>
18:25:45.103551 getsockname(25, {sa_family=AF_INET, sin_port=htons(7878), sin_addr=inet_addr("192.168.17.147")}, [16]) = 0 <0.000026>
18:25:45.103711 getpeername(25, {sa_family=AF_INET, sin_port=htons(50857), sin_addr=inet_addr("192.168.17.179")}, [16]) = 0 <0.000022>
18:25:45.103850 write(1, "REMOVE! 25 <socket.socket fd=25,"..., 148) = 148 <0.000039>
18:25:45.103979 sendfile(25, 24, [273672] => [403992], 478686) = 130320 <0.000060>
18:25:45.104165 getsockname(25, {sa_family=AF_INET, sin_port=htons(7878), sin_addr=inet_addr("192.168.17.147")}, [16]) = 0 <0.000018>
18:25:45.104260 getpeername(25, {sa_family=AF_INET, sin_port=htons(50857), sin_addr=inet_addr("192.168.17.179")}, [16]) = 0 <0.000019>
18:25:45.104367 write(1, "ADD 25 <socket.socket fd=25, fam"..., 144) = 144 <0.000026>
18:25:45.104473 epoll_ctl(6, EPOLL_CTL_ADD, 25, {EPOLLIN, {u32=4294967295, u64=18446744073709551615}}) = -1 EEXIST (File exists) <0.000017>
18:25:45.104541 epoll_ctl(6, EPOLL_CTL_DEL, 25, 0x7ffd90f368d0) = 0 <0.000023>
18:25:45.104601 ioctl(25, FIONBIO, [1]) = 0 <0.000016>
18:25:45.104678 epoll_ctl(6, EPOLL_CTL_ADD, 25, {EPOLLOUT, {u32=25, u64=25}}) = 0 <0.000022>
18:25:45.104769 epoll_wait(6, [{EPOLLOUT, {u32=25, u64=25}}], 1024, 93) = 1 <0.057713>
18:25:45.162769 getsockname(25, {sa_family=AF_INET, sin_port=htons(7878), sin_addr=inet_addr("192.168.17.147")}, [16]) = 0 <0.000030>
18:25:45.162926 getpeername(25, {sa_family=AF_INET, sin_port=htons(50857), sin_addr=inet_addr("192.168.17.179")}, [16]) = 0 <0.000026>
18:25:45.163065 write(1, "REMOVE! 25 <socket.socket fd=25,"..., 148) = 148 <0.000036>
18:25:45.163203 sendfile(25, 24, [403992] => [534312], 348366) = 130320 <0.000060>
18:25:45.163391 getsockname(25, {sa_family=AF_INET, sin_port=htons(7878), sin_addr=inet_addr("192.168.17.147")}, [16]) = 0 <0.000017>
18:25:45.163484 getpeername(25, {sa_family=AF_INET, sin_port=htons(50857), sin_addr=inet_addr("192.168.17.179")}, [16]) = 0 <0.000017>
18:25:45.163583 write(1, "ADD 25 <socket.socket fd=25, fam"..., 144) = 144 <0.000025>
18:25:45.163711 epoll_ctl(6, EPOLL_CTL_ADD, 25, {EPOLLIN, {u32=4294967295, u64=18446744073709551615}}) = -1 EEXIST (File exists) <0.000021>
18:25:45.163792 epoll_ctl(6, EPOLL_CTL_DEL, 25, 0x7ffd90f368d0) = 0 <0.000019>
18:25:45.163850 ioctl(25, FIONBIO, [1]) = 0 <0.000016>
18:25:45.163927 epoll_ctl(6, EPOLL_CTL_ADD, 25, {EPOLLOUT, {u32=25, u64=25}}) = 0 <0.000020>
18:25:45.163989 epoll_wait(6, [{EPOLLOUT, {u32=25, u64=25}}], 1024, 33) = 1 <0.017369>
18:25:45.181732 getsockname(25, {sa_family=AF_INET, sin_port=htons(7878), sin_addr=inet_addr("192.168.17.147")}, [16]) = 0 <0.000050>
18:25:45.181960 getpeername(25, {sa_family=AF_INET, sin_port=htons(50857), sin_addr=inet_addr("192.168.17.179")}, [16]) = 0 <0.000026>
18:25:45.182252 write(1, "REMOVE! 25 <socket.socket fd=25,"..., 148) = 148 <0.000049>
18:25:45.182423 sendfile(25, 24, [534312] => [664632], 218046) = 130320 <0.000133>
18:25:45.182797 getsockname(25, {sa_family=AF_INET, sin_port=htons(7878), sin_addr=inet_addr("192.168.17.147")}, [16]) = 0 <0.000027>
18:25:45.183088 getpeername(25, {sa_family=AF_INET, sin_port=htons(50857), sin_addr=inet_addr("192.168.17.179")}, [16]) = 0 <0.000164>
18:25:45.183530 write(1, "ADD 25 <socket.socket fd=25, fam"..., 144) = 144 <0.000143>
18:25:45.183813 epoll_ctl(6, EPOLL_CTL_ADD, 25, {EPOLLIN, {u32=4294967295, u64=18446744073709551615}}) = -1 EEXIST (File exists) <0.000022>
18:25:45.183929 epoll_ctl(6, EPOLL_CTL_DEL, 25, 0x7ffd90f368d0) = 0 <0.000167>
18:25:45.184256 ioctl(25, FIONBIO, [1]) = 0 <0.000048>
18:25:45.184526 epoll_ctl(6, EPOLL_CTL_ADD, 25, {EPOLLOUT, {u32=25, u64=25}}) = 0 <0.000142>
18:25:45.184805 epoll_wait(6, [{EPOLLOUT, {u32=25, u64=25}}], 1024, 13) = 1 <0.012150>
18:25:45.197446 getsockname(25, {sa_family=AF_INET, sin_port=htons(7878), sin_addr=inet_addr("192.168.17.147")}, [16]) = 0 <0.000051>
18:25:45.197749 getpeername(25, {sa_family=AF_INET, sin_port=htons(50857), sin_addr=inet_addr("192.168.17.179")}, [16]) = 0 <0.000114>
18:25:45.198159 write(1, "REMOVE! 25 <socket.socket fd=25,"..., 148) = 148 <0.000166>
18:25:45.198559 sendfile(25, 24, [664632] => [752358], 87726) = 87726 <0.000114>
18:25:45.198925 getsockname(25, {sa_family=AF_INET, sin_port=htons(7878), sin_addr=inet_addr("192.168.17.147")}, [16]) = 0 <0.000020>
18:25:45.199062 getpeername(25, {sa_family=AF_INET, sin_port=htons(50857), sin_addr=inet_addr("192.168.17.179")}, [16]) = 0 <0.000026>
18:25:45.199239 write(1, "CLOSE 25 <socket.socket fd=25, f"..., 146) = 146 <0.000049>
18:25:45.199385 write(1, "\n", 1) = 1 <0.000025>
18:25:45.199494 close(25) = 0 <0.000021>
18:25:45.199595 close(24) = 0 <0.000024>
18:25:45.200256 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2403, ...}) = 0 <0.000032>
18:25:45.200889 write(2, "[\33[34m2016-11-19 18:25:45,200\33[0"..., 333) = 333 <0.000031>
18:25:45.201266 epoll_wait(6, [], 1024, 0) = 0 <0.000036>
18:25:45.201492 epoll_ctl(6, EPOLL_CTL_ADD, 15, {EPOLLIN, {u32=4294967295, u64=18446744073709551615}}) = -1 EEXIST (File exists) <0.000025>
18:25:45.201601 epoll_ctl(6, EPOLL_CTL_DEL, 15, 0x7ffd90f39590) = 0 <0.000036>
18:25:45.201713 ioctl(15, FIONBIO, [1]) = 0 <0.000026>
18:25:45.201844 epoll_ctl(6, EPOLL_CTL_ADD, 15, {EPOLLIN, {u32=15, u64=15}}) = 0 <0.000026>
18:25:45.201949 epoll_wait(6, [], 1024, 0) = 0 <0.000018>
18:25:45.202011 epoll_wait(6, [{EPOLLOUT, {u32=25, u64=25}}], 1024, 81) = 1 <0.008192>
18:25:45.210303 epoll_ctl(6, EPOLL_CTL_DEL, 25, 0x7ffd90f37060) = -1 EBADF (Bad file descriptor) <0.000101>
18:25:45.210532 epoll_wait(6, [{EPOLLOUT, {u32=25, u64=25}}], 1024, 72) = 1 <0.000178>
18:25:45.210950 epoll_ctl(6, EPOLL_CTL_DEL, 25, 0x7ffd90f37060) = -1 EBADF (Bad file descriptor) <0.000159>
18:25:45.211247 epoll_wait(6, [{EPOLLOUT, {u32=25, u64=25}}], 1024, 63) = 1 <0.000064>
18:25:45.211420 epoll_ctl(6, EPOLL_CTL_DEL, 25, 0x7ffd90f37060) = -1 EBADF (Bad file descriptor) <0.000045>
18:25:45.211531 epoll_wait(6, [{EPOLLOUT, {u32=25, u64=25}}], 1024, 54) = 1 <0.000048>
18:25:45.211620 epoll_ctl(6, EPOLL_CTL_DEL, 25, 0x7ffd90f37060) = -1 EBADF (Bad file descriptor) <0.000018>
18:25:45.211675 epoll_wait(6, [{EPOLLOUT, {u32=25, u64=25}}], 1024, 45) = 1 <0.000022>
I could go into more detail about this but it's rather complex and I believe there are a number of issues you could tie back to the fact that remove_* isn't syncronous as it is with the stdlib implementation of EventLoop.
Using the asyncio example udp echo server with uvloop. Works perfectly without uvloop, but once uvloop is used, the remote addr isn't set on the transport. Fails to send packets to the remote machine.
Note: It works if you run the server/client on the same machine. To replicate this, try going between two machines.
Inspecting the socket of the created transport, I see <socket.socket fd=10, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 55216)>
.
With asyncio, raddr is set (<socket.socket fd=6, family=AddressFamily.AF_INET, type=2050, proto=17, laddr=('10.200.0.9', 33074), raddr=('10.10.8.30', 9999)>
).
I'd like to use this with aiosip, but I'm also seeing this problem there.
uvloop targets asyncio specifically but given that Tornado can use asyncio as the main event loop [1] [2] it looks like integrating the 3 (tornado + asyncio loop + uvloop) could be possible, and perhaps also fairly easily.
I haven't looked into uvloop code specifically but given that the tornado -> asyncio loop bridge is already in place in tornado's code, such an integration could already be possible in principle, in which case the proposal is to investigate how to "enable it" and document it (possibly in both projects).
Using uvloop and asyncio.SubprocessProtocol to run shell commands asynchronously, these create zombie processes which are not garbage collected properly unless you run asyncio.get_child_watcher().attach_loop(loop)
.
Since I am using uvloop as the event loop policy, for some reason it is not calling the unix_events function from the asyncio module and returning this error:
Traceback (most recent call last): File "/root/discord/bot0.py", line 18, in <module> bot = NotSoBot(loop=loop, shard_id=shard_id, shard_count=shard_count, dev_mode=dev_mode, max_messages=10000) File "/root/discord/bot.py", line 126, in __init__ asyncio.get_child_watcher().attach_loop(self.loop) File "/usr/lib/python3.5/asyncio/events.py", line 647, in get_child_watcher return get_event_loop_policy().get_child_watcher() File "/usr/lib/python3.5/asyncio/events.py", line 538, in get_child_watcher raise NotImplementedError NotImplementedError
The function is only available for unix systems which I am on but with uvloop it doesn't work.
wonderful project!when to suppert windows?
Or is it possible now?
Some of my timing- and debouncing-related tests are failing when I replace asyncio loop with uvloop. I tracked it down to loop.time
not increasing when the loop isn't running (e.g. when I'm blocking with time.sleep
).
First, an example of call_later
malfunctioning:
import asyncio
import time
import uvloop
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
def f():
print('f ', time.time())
async def main(loop):
await asyncio.sleep(0.001)
time.sleep(1)
print('main', time.time())
loop.call_later(1, f)
await asyncio.sleep(1)
loop = asyncio.get_event_loop()
loop.run_until_complete(main(loop))
There should be a 1-second difference between the two printed times, but actual output is
main 1463724375.72489
f 1463724375.725403
Second, a demonstration that this has something to do with loop.time
:
import asyncio
import time
import uvloop
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
async def test1(loop):
T0 = loop.time()
time.sleep(1)
print(loop.time() - T0)
async def test2(loop):
T0 = loop.time()
time.sleep(1)
await asyncio.sleep(0.001)
print(loop.time() - T0)
async def test3(loop):
T0 = loop.time()
await asyncio.sleep(0.001)
time.sleep(1)
print(loop.time() - T0)
loop = asyncio.get_event_loop()
loop.run_until_complete(test1(loop))
loop.run_until_complete(test2(loop))
loop.run_until_complete(test3(loop))
In all three cases a difference of 1 second should be printed. Actual output is
0.0
1.0019999999785796
0.0010000000474974513
Note that test2
works as expected because the loop is given a chance to "catch up" during that millisecond wait. test3
shows that asyncio.sleep
before the blocking call is accounted for.
Both examples behave "correctly" if you just comment out the set_event_loop_policy
call.
I have no idea how to fix this but I hope this report helps you. Really excited to try uvloop in a stock trading platform I work on!
I'm not sure that this is a problem uvloop.
aio-libs/aiohttp#925
import asyncio
import aiohttp
import uvloop
from aiohttp.resolver import AsyncResolver
async def run(_loop):
r = AsyncResolver(loop=_loop)
conn = aiohttp.TCPConnector(loop=_loop, resolver=r)
with aiohttp.ClientSession(connector=conn, loop=_loop) as ses:
async with ses.get('http://vk.com') as resp:
if resp.status == 200:
print(await resp.read())
else:
print(resp.status)
if __name__ == '__main__':
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
loop = asyncio.get_event_loop()
loop.run_until_complete(run(loop))
loop.close()
Result:
python3: src/unix/core.c:888: uv__io_stop: Assertion `loop->watchers[w->fd] == w' failed.
bash: line 1: 4020 aborted /usr/bin/python3 -u /vagrant/test.py
python finished with exit code 134
Found a spelling error in an error message at https://github.com/MagicStack/uvloop/blob/master/uvloop/handles/udp.pyx#L218.
Thanks for the great work!
Hi guys,
My program runs normally when using default asyncio policy. But if installing the uvloop event loop policy,
import uvloop
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
the following exception will be raised:
Traceback (most recent call last):
File "/opt/venv/websec/lib/python3.5/site-packages/aiohttp/client.py", line 565, in __aenter__
self._resp = yield from self._coro
File "/opt/venv/websec/lib/python3.5/site-packages/aiohttp/client.py", line 197, in _request
with Timeout(timeout, loop=self._loop):
File "/opt/venv/websec/lib/python3.5/site-packages/async_timeout/__init__.py", line 33, in __enter__
raise RuntimeError('Timeout context manager should be used '
RuntimeError: Timeout context manager should be used inside a task
Thanks!
Code work switching on cork/nodelay looks like this: https://github.com/KeepSafe/aiohttp/blob/master/aiohttp/parsers.py#L243
It relies on socket.setsockopt(...)
calls.
But uvloop returns another socket.
AFAIK socket flags are not shared between dups.
The problem is not such bad because libuv switches on nodelay by default IIRC but enabling/disabling CORK still makes sense.
Could uvloop be implemented using CFFI? How hard would it be to do so?
I ask because at some point in the future PyPy3 will reach compatibility with Python 3.5 (It could be a while, but PyPy3 3.3 is in alpha right now so there is hope :-) and it would be really great if uvloop were usable on it :-)
uvloop
breaks aiohttp's capability of serving static files. It makes the server send the response headers after the body.
To run this, have aiohttp and uvloop installed, then run the gist ( https://gist.github.com/diogobaeder/0558c43d785cd82a1ecff553bc583867 ):
$ python uvhttp.py
Then make a request to the app, making sure to get verbose output:
$ curl -v localhost:8765/jquery.js
These are the results I'm getting when serving jQuery:
Notice how with uvloop the response headers are sent after the body.
I'm not acquainted with the internals of aiohttp
or uvloop
, but it seems that the former relies on some parser from asyncio
and uvloop
seems to replace it by another one that behaves differently.
Any ideas of why this happens? At the moment this is preventing me to use uvloop
, unfortunately...
Thanks!
There appears to be a discrepancy in valid port
arguments to the event loop's create_server
method. With asyncio
I can use 0
and None
to obtain a server bound to an ephemeral port. With uvloop
it appears that None
is invalid and raises an error from the _getaddrinfo
method.
File "uvloop/loop.pyx", line 1026, in create_server (uvloop/loop.c:19287)
File "uvloop/loop.pyx", line 505, in uvloop.loop.Loop._getaddrinfo (uvloop/loop.c:11976)
TypeError: port must be a str, bytes or int
I came across this because in my use case I create many servers on ephemeral ports that then register their actual bound endpoint details with a service discovery mechanism. In my asyncio
based implementation I leave the port at its default value (e.g. port=None
) but this failed when I was testing switching over to uvloop
. I was able to modify my code to explicitly use port=0
to get it to work with both event loops.
It would be nice if uvloop
was a true drop in replacement for asyncio
. I expect that a check in the _getaddrinfo method could accomodate this discrepancy.
I have created a short gist that can be used to demonstrate the difference.
Is UVLoop production ready? Getting ready to implement some new microservices on top of asyncio using UVLoop.
When I deploy an aiohttp app with aiohttp.worker.GunicornWebWorker it looks like the loop is blocked and app never serves requests.
When I run without the worker (with web.run_app(app)
), it works well.
I think the problem could be caused by the GunicornWebWorker
itself since it's closing the default asyncio's loop and then creating new one. But I'm not sure.
Is this issue related to uvloop
or I should post to aiohttp
's issue tracker?
Hi,
Could you move build logic from Makefile to setup.py? Like in asyncpg's 93733dba6c09673660614c014bae955dda9e8796?
TIA :)
Tried running aiopyramid websocket example, it raises AttributeError: 'uvloop.loop.TCPTransport' object has no attribute '_protocol'
.
With asyncio event loop it's working fine.
aiopyramid is using internal API, but is there any way of providing similar _protocol
property in uvloop?
pip install uvloop throws this error
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-build-3p6l9uj5/uvloop/setup.py", line 81, in <module>
include_package_data=True
File "/usr/lib/python3.5/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/lib/python3.5/distutils/dist.py", line 955, in run_commands
self.run_command(cmd)
File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/home/anand/.virtualenvs/35/lib/python3.5/site-packages/setuptools/command/install.py", line 61, in run
return orig.install.run(self)
File "/usr/lib/python3.5/distutils/command/install.py", line 583, in run
self.run_command('build')
File "/usr/lib/python3.5/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/usr/lib/python3.5/distutils/command/build.py", line 135, in run
self.run_command(cmd_name)
File "/usr/lib/python3.5/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/home/anand/.virtualenvs/35/lib/python3.5/site-packages/setuptools/command/build_ext.py", line 49, in run
_build_ext.run(self)
File "/usr/lib/python3.5/distutils/command/build_ext.py", line 338, in run
self.build_extensions()
File "/tmp/pip-build-3p6l9uj5/uvloop/setup.py", line 40, in build_extensions
self.build_libuv()
File "/tmp/pip-build-3p6l9uj5/uvloop/setup.py", line 35, in build_libuv
subprocess.run(['make', j_flag], cwd=LIBUV_DIR, env=env, check=True)
File "/usr/lib/python3.5/subprocess.py", line 711, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['make', '-j4']' returned non-zero exit status 2
----------------------------------------
in my project paws I use an uncommon usage of loop.create_server
by passing a pre-created reusable non-blocking socket that may be causing an issue down the line with uvloop as shown in this screenshot.
I'm not 100% certain of this, but is this just an unhanded path or an incompatibility?
With release 0.4.11, if you leave host=None while setting the port with create_sever() you end up with:
File "uvloop/loop.pyx", line 1028, in create_server (uvloop/loop.c:19317)
File "uvloop/loop.pyx", line 511, in uvloop.loop.Loop._getaddrinfo (uvloop/loop.c:12089)
TypeError: host must be a str or bytes
This is different behavior from the base asyncio create_server that allows host=None while setting the port.
I picked this up in my unit tests which are admittedly testing some edge cases. When trying to use the loop to register a signal handler for SIGKILL (which itself is invalid) asyncio
and uvloop
behave differently.
A simple test that demonstrates the difference is this:
import asyncio
import signal
# import uvloop
# asyncio.DefaultEventLoopPolicy = uvloop.EventLoopPolicy
# asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
def my_sig_handler():
pass
loop = asyncio.get_event_loop()
loop.add_signal_handler(signal.SIGKILL, my_sig_handler)
loop.stop()
loop.close()
Running the script as is produces the asyncio
output:
$ python3.5 uvloop-sig-test.py
Traceback (most recent call last):
File "/usr/local/lib/python3.5/asyncio/unix_events.py", line 95, in add_signal_handler
signal.signal(sig, _sighandler_noop)
File "/usr/local/lib/python3.5/signal.py", line 47, in signal
handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
OSError: [Errno 22] Invalid argument
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "uvloop-sig-test.py", line 31, in <module>
loop.add_signal_handler(signal.SIGKILL, my_sig_handler)
File "/usr/local/lib/python3.5/asyncio/unix_events.py", line 108, in add_signal_handler
raise RuntimeError('sig {} cannot be caught'.format(sig))
RuntimeError: sig 9 cannot be caught
By uncommenting the uvloop part and running the script will produce the uvloop output:
$ python3.5 uvloop-sig-test.py
Traceback (most recent call last):
File "uvloop-sig-test.py", line 31, in <module>
loop.add_signal_handler(signal.SIGKILL, my_sig_handler)
File "uvloop/loop.pyx", line 1632, in uvloop.loop.Loop.add_signal_handler (uvloop/loop.c:30442)
File "uvloop/handles/signal.pyx", line 52, in uvloop.loop.UVSignal.start (uvloop/loop.c:42667)
File "uvloop/handles/handle.pyx", line 145, in uvloop.loop.UVHandle._fatal_error (uvloop/loop.c:37619)
OSError: [Errno 22] Invalid argument
On the box that I ran this test on the Python version was 3.5.0 (not the latest 3.5.1) so the Python module line numbers may be a little out.
I actually think that uvloop
does the right thing in reporting the first error. However it would appear that the asyncio
developers observed that the error reported is not very informative and wrapped it with a more informative error report.
asyncio
event loop has useful property BaseEventLoop.slow_callback_duration
for logging long blocking operations [1]. It is not most critical feature but nice to have it during development.
[1] https://docs.python.org/3.5/library/asyncio-dev.html#debug-mode-of-asyncio
AFAIK, libuv also support async filesystem calls. Can this be implemented in uvloop (or based on uvloop) with nice await with
and some pathlib
awaitable coroutines? If so, «not good» executors for filesystem calls can be dropped 👏
Consider providing wheels for all the major platforms. They install much more quickly and do no require your users to have a C toolchain and development libs installed.
Aiohttp recomended deploy Gunicorn + Nginx
Gunicorn (aiohttp.worker.GunicornWebWorker) + unix socket Nginx - OK
Ginicorn (aiohttp.worker.GunicornUVLoopWebWorker) + unix sockets Nginx - error
Traceback (most recent call last):
File "/usr/local/python/3.5.2/lib/python3.5/site-packages/aiohttp/helpers.py", line 407, in log
[message, environ, response, transport, time]))
File "/usr/local/python/3.5.2/lib/python3.5/site-packages/aiohttp/helpers.py", line 394, in _format_line
return tuple(m(args) for m in self._methods)
File "/usr/local/python/3.5.2/lib/python3.5/site-packages/aiohttp/helpers.py", line 394, in
return tuple(m(args) for m in self._methods)
File "/usr/local/python/3.5.2/lib/python3.5/site-packages/aiohttp/helpers.py", line 347, in _format_a
peername = args[3].get_extra_info('peername')
File "uvloop/handles/tcp.pyx", line 150, in uvloop.loop.TCPTransport.get_extra_info (uvloop/loop.c:66876)
File "uvloop/dns.pyx", line 60, in uvloop.loop.__convert_sockaddr_to_pyaddr (uvloop/loop.c:82997)
RuntimeError: cannot convert sockaddr into Python object
With asyncio default event loop the following works fine:
import asyncio
async def async_gen():
yield 5
async def run():
val = await async_gen().__anext__()
print(val)
asyncio.get_event_loop().run_until_complete(run())
With uvloop, the following will raise a AttributeError: 'Loop' object has no attribute '_write_to_self'
import asyncio
import uvloop
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
async def async_gen():
yield 5
async def run():
val = await async_gen().__anext__()
print(val)
asyncio.get_event_loop().run_until_complete(run())
After a quick glance it looks like some of the new async generator code is calling _write_to_self
, in particular the _asyncgen_finalizer_hook
method. However it doesn't seem like the uvloop event loop has implemented that method yet.
This is not an issue but perhaps a request to maybe expand on documentation on how to use uvloop with other frameworks. The performance on it is very impressive, however, it's so low level, taking this into a production application would be quite a challenge.
What would be the best practice approach to doing this? Combine with tornado or any other async libary? Leverage other smaller libraries to handle things like user authentication and http parsing. Or does uvloop just need these features introduced directly into it to maintain the performance it currently has?
If I am using the wrong forum for asking this question I apologize and happy to move this discussion somewhere else.
Add support for preexec_fn
in loop.subprocess_exec
-- a callback that gets called in the forked process right before exec
.
Some ToDo items we need completed before we make an alpha release:
loop.create_connection
loop.create_server
to increase the coverageloop.create_unix_server
loop.create_unix_connection
loop.subprocess_shell
and loop.subprocess_exec
loop.getnameinfo
loop.add_signal_handler
and loop.remove_signal_handler
loop.connect_read_pipe
and loop.connect_write_pipe
loop.create_datagram_endpoint
Would love to look at adopting uvloop in a number of high-scale tornado apps we have in production. Those are running Python 2.7 - what's stopping this library from adding support?
After performing STARTTLS, transp.get_extra_info("socket")
still returns the original socket and not the SSL socket. In the builtin asyncio event loop the transports extra info is upgraded to point to the SSL socket. I am unsure if this happens after a direct TLS connection or not.
I also don't have an easy to reproduce example without requiring an account somewhere; using the latest version of slixmpp to connect to any XMPP server should attempt STARTTLS and reproduce the issue, but I'm afraid that's not the most helpful advice in a bug report. I'll try to put together a minimal working example.
When using the built-in event loop, you need to do something like this:
server = await asyncio.start_unix_server(client_connected, path=path)
try:
# ...
finally:
server.close()
await server.wait_closed()
os.unlink(path)
However, when I use uvloop, the os.unlink()
call raises a FileNotFoundError
exception.
I was wondering if you already know about this and if it is intentional?
I don't mind adding a try/except clause to silence the exception, but if you're aiming for this event loop to be a drop-in replacement for the built-in event loop, I guess you're going to want to change this?
I just rebuild my wormhole proxy docker image today (Dockerfile). After I use it for a few minutes, it crashed and show error messages like these (randomly on each time it crashed).
python3.5: src/unix/core.c:236: uv__finish_close: Assertion `handle->flags & UV_CLOSING' failed.
or
python3.5: src/unix/core.c:264: uv__finish_close: Assertion `0' failed.
or
python3.5: src/unix/stream.c:444: uv__stream_destroy: Assertion `(((const QUEUE ) (&(stream->loop)->active_reqs) == (const QUEUE *) ((QUEUE *) &(((&(stream->loop)->active_reqs))[0]))) == 0)' failed.
I also tried to build it on Alpine Linux image (Dockerfile) the result is also the same.
BTW, I have no problem with the previous image that was built around 2 weeks ago, the uvloop version on that image was 0.4.30.
Hi, I'm testing uvloop integration with pulsar and it works a treat!
However, I have one problem when running in multiprocessing mode.
If I Ctrl-C on the shell, the master process receives the SIGINT signal but the subprocesses do not and do not respond to the master process sending QUIT signals either.
The subprocesses get eventually killed by pulsar master process using SIGKILL.
This only happen with uvloop.
Is there something I need to be aware about Ctrl-C with uvloop?
Code:
import asyncio
from aiohttp import web
import uvloop
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
async def handle(request):
name = request.match_info.get('name', "Anonymous")
text = "Hello, " + name
return web.Response(text=text)
app = web.Application()
app.router.add_get('/', handle)
app.router.add_get('/{name}', handle)
web.run_app(app)
traceback:
Exception in callback Task._step
handle: <Handle Task._step>
Traceback (most recent call last):
File "uvloop/cbhandles.pyx", line 63, in uvloop.loop.Handle._run (uvloop/loop.c:38343)
File "uvloop/task.pyx", line 159, in uvloop.loop.BaseTask._fast_step (uvloop/loop.c:99783)
AttributeError: '_GatheringFuture' object has no attribute '_blocking'
We are seeing a serious memory leak that only shows up when using UVLoop on Gunicorn. Examining the GC state does not show anything useful, which indicates to me that it is likely inside native code, so I report this here. Under moderate load, we see each Gunicorn worker taking 2GB ram within 18-24 hours.
We are presently retesting under 0.4.32 and will update if we still see it. If you have any suggestions for debugging it that may be helpful, that would be useful too. We only see this leak under UVLoop and not the default asyncio loop, so it has to be related to UVLoop.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.