Assertion `loop->watchers[w->fd] == w' failed about pyuv HOT 19 CLOSED

Multiplexing events? In uv-unix, that's fairly easy to do. Not so in uv-win, apparently, but Bert said he'll take patches.

I'm not against bringing the old behavior back but only if it's consistent on all platforms.

I won't be able to send patches, but I would like to point out that it is a real pain that libuv can only have one watcher per FD. I have cursed on libuv a couple of times :)

Two examples.

1. When integration an event loop into libdbus, the callback API expects that it can set multiple watches per FD. In practice, it only sets two watcher per FD: a reader and a writer (in separate calls).
2. The new Python EventLoop (PEP3156) has separate add_reader() and add_writer() functions. So i need to track the FDs that are allocated when implementing this API for libuv here: https://github.com/geertj/looping

from pyuv.

pyuv is not abort()-ing, libuv is. The problem is that you are using the same fd on 2 Poll handles, which is unsupported, and apparently crashes when you stop the second one :-(. I agree that getting a crash in this case is not nice, so could you please open a bug report in the libuv project and suggest them to give a proper error is there is already a uv_poll_t handle with a given fd? Thanks!

from pyuv.

I know it's libuv that is calling abort(). I thought maybe you could see if you can keep some extra state in pyuv to make sure you don't allow the Python user to put the C library in an unsupported state.

from pyuv.

I see. Well, I don't want to do extra bookkeeping in pyuv if it can be
fixed in libuv :-)

I'll try to fix this myself and submit a pull request, but it will take me
a bit of time.

On Friday, December 28, 2012, Geert Jansen wrote:

I know it's libuv that is calling abort(). I thought maybe you could see
if you can keep some extra state in pyuv to make sure you don't allow the
Python user to put the C library in an unsupported state.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/54#issuecomment-11743394.

/Saúl
http://saghul.net | http://sipdoc.net

from pyuv.

I raised the issue upstream.

from pyuv.

As a background: I had similar issues with libdbus and my wrapper "python-dbusx". For "security" reasons, libdbus decides to abort() on almost anything, including invalid input. Since this is highly non-standard behavior for Python, in my case I decided to check all input myself in the extension module, and raise an exception before calling into libdbus in case of invalid input.

from pyuv.

Just another thought.. Maybe libuv could allow you to install an "assertion failed" handler. That way you don't need to find all the 100s of places where it has assertions and ensure they cannot happen.

from pyuv.

Maybe libuv could allow you to install an "assertion failed" handler.

And then what? Libuv is still in an unexpected state. You can't go on and pretend nothing happened.

from pyuv.

@bnoordhuis yeah, I don't like that idea either. Is there a way to get the old behavior back? That is, allow more than one handle for a given fd, even if there are potential problems in Windows?

from pyuv.

Is there a way to get the old behavior back? That is, allow more than one handle for a given fd, even if there are potential problems in Windows?

Multiplexing events? In uv-unix, that's fairly easy to do. Not so in uv-win, apparently, but Bert said he'll take patches.

I'm not against bringing the old behavior back but only if it's consistent on all platforms.

from pyuv.

And then what? Libuv is still in an unexpected state. You can't go on and pretend nothing happened.

Raise an AssertionError. Still not ideal but a lot better than calling abort(). At least, the Python programmer will get a backtrace and see where things went wrong. (Yes: i'm implying setjmp()/longjmp() in the assertion handler).

from pyuv.

When I find some time I'll add some bookkeeping with a hash table. I found this one, for reference: http://uthash.sourceforge.net/ The idea is simple (I hope :) ): check if the fd is there before init and remove it just after calling close.

from pyuv.

When integration an event loop into libdbus, the callback API expects that it can set multiple watches per FD. In practice, it only sets two watcher per FD: a reader and a writer (in separate calls).

This is also what gevent does, so currently uvent is broken :-(

The new Python EventLoop (PEP3156) has separate add_reader() and add_writer() functions. So i need to track the FDs that are allocated when implementing this API for libuv here: https://github.com/geertj/looping

Heh, I'm also working on an implementation of that PEP ;-)

from pyuv.

Heh, I'm also working on an implementation of that PEP ;-)

Interesting.. The right place for this would for sure be part of pyuv and not as a standalone package. I should put that in the README. If it helps you out, you can take any code from this and I can license it to you in any way you want (although I assume you will be doing this in C?).

The only reason I created this as an external module is because I needed something urgently for this project:

https://github.com/geertj/python-dbusx

Regarding uvent.. I tried to do something similar but i wanted not to depend on gevent and make something a lot simpler. Currently it's mostly a proof of concept:

https://github.com/geertj/gruvi

It is a lot simpler (< 700 lines vs 9000 for gevent) but does normal sockets and ssl already. I am not sure yet where I am going to take this.

from pyuv.

Interesting.. The right place for this would for sure be part of pyuv and not as a standalone package. I should put that in the README. If it helps you out, you can take any code from this and I can license it to you in any way you want (although I assume you will be doing this in C?).

Not really, pyuv is libuv + Python, that's it :-) I'm working on a tulip-uv thing. The event loop works on Py2-3, but the tasks can only work on Py3 due to yield from...

Thanks for the offer, I'll check it out since I'm preparing it for a talk at FOSDEM.

Regarding uvent.. I tried to do something similar but i wanted not to depend on gevent and make something a lot simpler. Currently it's mostly a proof of concept:

I'll be sure to check that one too! Funny enough I also have an ongoing project similar to gevent, but I started from eventlet instead :-)

from pyuv.

I'll be sure to check that one too! Funny enough I also have an ongoing project similar to gevent, but I started from eventlet instead :-)

Funny indeed. Is there a better place to discuss this than an issue tracker? It's a interesting topic. With libraries like pyuv and greenlet, the "glue layer" can be very small in my view.

from pyuv.

Sure! shoot me an email! saghul (at) gmail (dot) com

from pyuv.

I'm really trying to figure out how to use libcurl (pycurl) w/python, in a nice event loop. Have been at it for quite some time w/different implementations.

Is the issue HERE (THIS thread) still relevant?

My Program crashes w/:

  python: src/unix/core.c:856: uv__io_stop: Assertion `loop->watchers[w->fd] == w' failed.

I read in another thread that it happens during garbage collection, which might explain while It's not happening for me when going through a proxy..

Is the 'problem' OS-related/i.e. not program related. Meaning, epoll does not support two 'watchers' for a file descriptor... and (from my understanding), libuv removed some of the 'excess' of pyev, that introduces this problem?

I'm just starting to be able to read C and have a semi-understanding of whats going on.. if I were able to have libcurl and libuv 'looking' at the same poll hash (don't know if this is right..)--would this fix the problem below?

 https://github.com/pycurl/pycurl/issues/324

any feedback/insight is very much appreciated.

from pyuv.

Yes it is. This limitation still exists, so you have to be careful not to create 2 Poll handles for the same fd or libuv will abort.

from pyuv.