0rpc / zerorpc-python Goto Github PK

I'd be helpful and informative to have a document describing the protocol. It'd also help when people want to implement ZeroRPC in other languages.

ZeroRPC 0.4.4 depends on pyzmq==13.1.0. Python/setuptools don't handle "==" dependencies well - if a project depends on both pyzmq and zerorpc, and setuptools decides to download pyzmq first, it will install the newest pyzmq, and then fail when it reaches the installation of zerorpc...
I understand that the newest pyzmq breaks zerorpc, but I'm not sure what's the problem. Is it possible to adapt zerorpc and/or open an issue for pyzmq to fix their issues?

Now that we have ZeroRPC bindings in multiple languages, it would make sense for the CLI to be put in a separate repo

Hi
I'm a trying to expose an object. (that I didn't build) but zerorpc fails with an AttributeError complaining about name.

As I have looked at the code I know that the problem is caused because my object's super class has a code like this (simplified):

def get_connection(self):
        if self.con is None:
            self.con = sqlite.connect()
        return self.con

connection = property(get_connection, None)

So I wonder, zerorpc can only expose objects that only have methods.
If that is the case I know that I can wrap my object in other that only has the methods so zerorpc can work.
Want to know what is common practice in zerorpc.

I followed the instructions to install zerorpc, but when I ran the Hello World examples, I encountered issue 56. Would have helped to have some documented warning.

I spent most part of the last two days trying to figure out why the Publisher/Subscriber pattern wasn't working.

After running pip install -I it worked on the first run.

First of all, thanks very much for this library. It's really useful and works nicely most of the time. However, I did encounter an issue when trying to expose a list subclass as a zeroservice. The initial exception was:

File "/Users/prak/code/venv/twola/lib/python2.7/site-packages/zerorpc/core.py", line 109, in _inject_builtins
self._methods['_zerorpc_list'] = lambda: [m for m in self._methods
TypeError: list indices must be integers, not str

However, this was just a symptom caused by the fact that ServerBase._filter_methods (core.py:71) seems to try and check if its methods argument is a dict by seeing if it has a getitem method. I was able to get by this explicitly doing isinstance(dict).

Next there was an issue with the dict comprehension in the return statement (core.py:76) in that checking whether a method is "not in server_methods" requires that thing to be hashable which means built in functions break it. A fix is to replace the return call with:

    return dict((k, getattr(methods, k))
            for k in dir(methods)
            if callable(getattr(methods, k))
            and not k.startswith('_')
            and (inspect.isbuiltin(getattr(methods, k)) or
                 getattr(methods, k) not in server_methods)
            )

which requires importing the inspect module. I'm not sure if this solution is worthwhile though because remoting dict subclasses will remain an issue and it really seems like it's the multiple calls to ServerBase._filter_methods from zerorpc.Server that are a bit awkward.

A test to reproduce the error (I put it in test_server.py):

def test_server_can_wrap_list():
    endpoint = random_ipc_endpoint()

    l = list()
    srv = zerorpc.Server(list)
    srv.bind(endpoint)

The test is pretty weak but I'm not familiar enough with the codebase yet to do much more. If you guys are ok with this solution, I'd be happy to go through the fork, commit, pull request process.

Is the code mentioned in @jpetazzo 's talk at PyCon about the dotcloud API cli which uses HTTP-> ZeroRPC available somewhere?

The zerorpc helper is very helpful (eh!), and actually, I start 99% of the ZeroRPC servers that I write with zerorpc --server, rather than writing a custom script.

However, I also want to do the following:

• enable a specific middleware;
• configure logging.

I currently work around the problem by enabling middlewares and setting up logging within the module loaded by zerorpc --server; but this leads to code repetition, and is therefore a Bad Thing.

I would love to have (at least) one of the following options:

• the zerorpc helper would automatically try to import a module named zerorpc_settings (and fail silently if this raises an ImportError);
• the zerorpc helper would automatically try to import the module indicated by the environment variable ZERORPC_SETTINGS_MODULE (if it is set);
• the zerorpc helper would allow an optional command-line flag, --settings, indicating the name of the module to import;
• if just importing a module feels weak, we could specify a callable instead (and the environment variable would become ZERORPC_SETTINGS_CALLABLE).

I'm willing to implement any of those, as long as I know that it has a chance to be merged (provided that the implementation isn't ugly nor kill baby seals).

What do you think, @bombela ?

In my environment this command from the first example in the README fails:

zerorpc --client --connect tcp://*:1234 strftime %Y/%m/%d

It works fine if the "*" is replaced with "localhost".

Environment:
Mac OS X 10.6.8
Python 2.7.3 (MacPorts)
zerorpc 0.3.0
pyzmq_static 2.2

Traceback:

connecting to "tcp://*:1234"
Traceback (most recent call last):
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/zerorpc", line 274, in
exit(main())
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/zerorpc", line 265, in main
return run_client(args)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/zerorpc", line 207, in run_client
setup_links(args, client)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/zerorpc", line 94, in setup_links
socket.connect(endpoint)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/zerorpc/socket.py", line 40, in connect
return self._events.connect(endpoint, resolve)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/zerorpc/events.py", line 212, in connect
r.append(self.socket.connect(endpoint))
File "socket.pyx", line 475, in zmq.core.socket.Socket.connect (zmq/core/socket.c:4508)
zmq.core.error.ZMQError: Invalid argument

I have an application for Avahi like: http://avahi.org/wiki/PythonBrowseExample

And I want zeroRPC in the same application. But when I start the server with the run method. There is an exception "LoopExit: This operation would block forever".

I assume this is because of mutliple main loops or something. I am very new to gevent etc.

How can I combine those two loops?

Servers are documented, but clients aren't.

Hey, Just curious how you're going to license it, I couldn't find anything in the headers or setup.py or a LICENSE file. For those reasons I quit reading since I'm working on something similar for a client right now.

self should not be one of the arguments since it's implicitly provided. This will be more consistent with the behavior of zerorpc-node.

Traceback (most recent call last):
  File "/var/tmp/izbox/mock/izbox-mainline/eggs/zerorpc-0.4.3.1-py2.7.egg/zerorpc/core.py", line 148, in _async_task
    functor.pattern.process_call(self._context, bufchan, event, functor)
  Local variables:
    'initial_event': <zerorpc.events.Event object at 0x14912cf8>
    'exc_infos': <repr truncated: <list object at 0x154cf638>>
    'self': <izbox.events.processor.EventsProcessorServer object at 0x388ec10>
    'bufchan': <zerorpc.channel.BufferedChannel object at 0x13624690>
    'functor': <zerorpc.decorators.rep object at 0x3891790>
    'protocol_v1': False
    'hbchan': <zerorpc.heartbeat.HeartBeatOnChannel object at 0x145315d0>
    'event': <zerorpc.events.Event object at 0x14912cf8>
    'channel': <zerorpc.channel.Channel object at 0x14757210>
  File "/var/tmp/izbox/mock/izbox-mainline/eggs/zerorpc-0.4.3.1-py2.7.egg/zerorpc/patterns.py", line 34, in process_call
    bufchan.emit_event(rep_event)
  Local variables:
    'rep_event': <zerorpc.events.Event object at 0x13be6680>
    'self': <zerorpc.patterns.ReqRep instance at 0x22b84d0>
    'bufchan': <zerorpc.channel.BufferedChannel object at 0x13624690>
    'functor': <zerorpc.decorators.rep object at 0x3891790>
    'result': <repr truncated: <dict object at 0x163103f0>>
    'context': <zerorpc.context.Context object at 0x28ea668>
    'req_event': <zerorpc.events.Event object at 0x14912cf8>
  File "/var/tmp/izbox/mock/izbox-mainline/eggs/zerorpc-0.4.3.1-py2.7.egg/zerorpc/channel.py", line 244, in emit_event
    self._channel.emit_event(event)
  Local variables:
    'self': <zerorpc.channel.BufferedChannel object at 0x13624690>
    'block': True
    'timeout': None
    'event': <zerorpc.events.Event object at 0x13be6680>
  File "/var/tmp/izbox/mock/izbox-mainline/eggs/zerorpc-0.4.3.1-py2.7.egg/zerorpc/heartbeat.py", line 111, in emit_event
    self._channel.emit_event(event)
  Local variables:
    'self': <zerorpc.heartbeat.HeartBeatOnChannel object at 0x145315d0>
    'event': <zerorpc.events.Event object at 0x13be6680>
  File "/var/tmp/izbox/mock/izbox-mainline/eggs/zerorpc-0.4.3.1-py2.7.egg/zerorpc/channel.py", line 160, in emit_event
    self._multiplexer.emit_event(event, self._zmqid)
  Local variables:
    'self': <zerorpc.channel.Channel object at 0x14757210>
    'event': <zerorpc.events.Event object at 0x13be6680>
  File "/var/tmp/izbox/mock/izbox-mainline/eggs/zerorpc-0.4.3.1-py2.7.egg/zerorpc/channel.py", line 67, in emit_event
    return self._events.emit_event(event, identity)
  Local variables:
    'self': <zerorpc.channel.ChannelMultiplexer object at 0x388e050>
    'event': <zerorpc.events.Event object at 0x13be6680>
    'identity': ['\x00h\x9d\xe5h']
  File "/var/tmp/izbox/mock/izbox-mainline/eggs/zerorpc-0.4.3.1-py2.7.egg/zerorpc/events.py", line 240, in emit_event
    parts.extend(['', event.pack()])
  Local variables:
    'self': <zerorpc.events.Events object at 0x3869ad0>
    'parts': ['\x00h\x9d\xe5h']
    'event': <zerorpc.events.Event object at 0x13be6680>
    'identity': ['\x00h\x9d\xe5h']
  File "/var/tmp/izbox/mock/izbox-mainline/eggs/zerorpc-0.4.3.1-py2.7.egg/zerorpc/events.py", line 140, in pack
    return msgpack.Packer().pack((self._header, self._name, self._args))
  Local variables:
    'self': <zerorpc.events.Event object at 0x13be6680>
  File "_packer.pyx", line 182, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:182)
  Local variables:
    'PackValueError': <class 'msgpack.exceptions.PackValueError'>
    '__builtins__': <module '__builtin__' (built-in)>
    '__file__': <repr truncated: <str object at 0x222e930>>
    '__package__': 'msgpack'
    'Packer': <type 'msgpack._packer.Packer'>
    '__test__': {}
    'packb': <built-in function packb>
    '__name__': 'msgpack._packer'
    '__doc__': None
    'pack': <built-in function pack>
  File "_packer.pyx", line 184, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:184)
  Local variables:
    'PackValueError': <class 'msgpack.exceptions.PackValueError'>
    '__builtins__': <module '__builtin__' (built-in)>
    '__file__': <repr truncated: <str object at 0x222e930>>
    '__package__': 'msgpack'
    'Packer': <type 'msgpack._packer.Packer'>
    '__test__': {}
    'packb': <built-in function packb>
    '__name__': 'msgpack._packer'
    '__doc__': None
    'pack': <built-in function pack>
  File "_packer.pyx", line 173, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:173)
  Local variables:
    'PackValueError': <class 'msgpack.exceptions.PackValueError'>
    '__builtins__': <module '__builtin__' (built-in)>
    '__file__': <repr truncated: <str object at 0x222e930>>
    '__package__': 'msgpack'
    'Packer': <type 'msgpack._packer.Packer'>
    '__test__': {}
    'packb': <built-in function packb>
    '__name__': 'msgpack._packer'
    '__doc__': None
    'pack': <built-in function pack>
  File "_packer.pyx", line 173, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:173)
  Local variables:
    'PackValueError': <class 'msgpack.exceptions.PackValueError'>
    '__builtins__': <module '__builtin__' (built-in)>
    '__file__': <repr truncated: <str object at 0x222e930>>
    '__package__': 'msgpack'
    'Packer': <type 'msgpack._packer.Packer'>
    '__test__': {}
    'packb': <built-in function packb>
    '__name__': 'msgpack._packer'
    '__doc__': None
    'pack': <built-in function pack>
  File "_packer.pyx", line 159, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:159)
  Local variables:
    'PackValueError': <class 'msgpack.exceptions.PackValueError'>
    '__builtins__': <module '__builtin__' (built-in)>
    '__file__': <repr truncated: <str object at 0x222e930>>
    '__package__': 'msgpack'
    'Packer': <type 'msgpack._packer.Packer'>
    '__test__': {}
    'packb': <built-in function packb>
    '__name__': 'msgpack._packer'
    '__doc__': None
    'pack': <built-in function pack>
  File "_packer.pyx", line 173, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:173)
  Local variables:
    'PackValueError': <class 'msgpack.exceptions.PackValueError'>
    '__builtins__': <module '__builtin__' (built-in)>
    '__file__': <repr truncated: <str object at 0x222e930>>
    '__package__': 'msgpack'
    'Packer': <type 'msgpack._packer.Packer'>
    '__test__': {}
    'packb': <built-in function packb>
    '__name__': 'msgpack._packer'
    '__doc__': None
    'pack': <built-in function pack>
  File "_packer.pyx", line 167, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:167)
  Local variables:
    'PackValueError': <class 'msgpack.exceptions.PackValueError'>
    '__builtins__': <module '__builtin__' (built-in)>
    '__file__': <repr truncated: <str object at 0x222e930>>
    '__package__': 'msgpack'
    'Packer': <type 'msgpack._packer.Packer'>
    '__test__': {}
    'packb': <built-in function packb>
    '__name__': 'msgpack._packer'
    '__doc__': None
    'pack': <built-in function pack>
  File "_packer.pyx", line 173, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:173)
  Local variables:
    'PackValueError': <class 'msgpack.exceptions.PackValueError'>
    '__builtins__': <module '__builtin__' (built-in)>
    '__file__': <repr truncated: <str object at 0x222e930>>
    '__package__': 'msgpack'
    'Packer': <type 'msgpack._packer.Packer'>
    '__test__': {}
    'packb': <built-in function packb>
    '__name__': 'msgpack._packer'
    '__doc__': None
    'pack': <built-in function pack>
  File "_packer.pyx", line 159, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:159)
  Local variables:
    'PackValueError': <class 'msgpack.exceptions.PackValueError'>
    '__builtins__': <module '__builtin__' (built-in)>
    '__file__': <repr truncated: <str object at 0x222e930>>
    '__package__': 'msgpack'
    'Packer': <type 'msgpack._packer.Packer'>
    '__test__': {}
    'packb': <built-in function packb>
    '__name__': 'msgpack._packer'
    '__doc__': None
    'pack': <built-in function pack>
  File "_packer.pyx", line 173, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:173)
  Local variables:
    'PackValueError': <class 'msgpack.exceptions.PackValueError'>
    '__builtins__': <module '__builtin__' (built-in)>
    '__file__': <repr truncated: <str object at 0x222e930>>
    '__package__': 'msgpack'
    'Packer': <type 'msgpack._packer.Packer'>
    '__test__': {}
    'packb': <built-in function packb>
    '__name__': 'msgpack._packer'
    '__doc__': None
    'pack': <built-in function pack>
  File "_packer.pyx", line 159, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:159)
  Local variables:
    'PackValueError': <class 'msgpack.exceptions.PackValueError'>
    '__builtins__': <module '__builtin__' (built-in)>
    '__file__': <repr truncated: <str object at 0x222e930>>
    '__package__': 'msgpack'
    'Packer': <type 'msgpack._packer.Packer'>
    '__test__': {}
    'packb': <built-in function packb>
    '__name__': 'msgpack._packer'
    '__doc__': None
    'pack': <built-in function pack>
  File "_packer.pyx", line 159, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:159)
  Local variables:
    'PackValueError': <class 'msgpack.exceptions.PackValueError'>
    '__builtins__': <module '__builtin__' (built-in)>
    '__file__': <repr truncated: <str object at 0x222e930>>
    '__package__': 'msgpack'
    'Packer': <type 'msgpack._packer.Packer'>
    '__test__': {}
    'packb': <built-in function packb>
    '__name__': 'msgpack._packer'
    '__doc__': None
    'pack': <built-in function pack>
  File "_packer.pyx", line 159, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:159)
  Local variables:
    'PackValueError': <class 'msgpack.exceptions.PackValueError'>
    '__builtins__': <module '__builtin__' (built-in)>
    '__file__': <repr truncated: <str object at 0x222e930>>
    '__package__': 'msgpack'
    'Packer': <type 'msgpack._packer.Packer'>
    '__test__': {}
    'packb': <built-in function packb>
    '__name__': 'msgpack._packer'
    '__doc__': None
    'pack': <built-in function pack>
  File "_packer.pyx", line 124, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:124)
  Local variables:
    'PackValueError': <class 'msgpack.exceptions.PackValueError'>
    '__builtins__': <module '__builtin__' (built-in)>
    '__file__': <repr truncated: <str object at 0x222e930>>
    '__package__': 'msgpack'
    'Packer': <type 'msgpack._packer.Packer'>
    '__test__': {}
    'packb': <built-in function packb>
    '__name__': 'msgpack._packer'
    '__doc__': None
    'pack': <built-in function pack>
OverflowError: long too big to convert

The traceback is thrown by gevent, and there's no wait to tell what exactly caused the exception.

I'm working with zerorpc, but need to be able to set some zmq socket options. It don't see a way of doing this without reaching into SocketBase._events._socket to call setsockopt. I'd rather not do that in case the non-public members are changed in future versions.

I'd be happy to implement this, but need a little guidance since I'm a bit new to zmq. This could be implemented as either part of SocketBase or as part of the context. If part of the context, it would ensure that all subsequent sockets are created with these options. Looking at the context implementation, maybe a create_socket middleware, combined with an override of Conext.socket(), could work.

Like I said, I'm fairly new to the project and zmq. I'm going to fork and start hacking on the middleware, but please let me know what makes the most sense.

pyzmq rolled pyzmq-static into itself meaning pyzmq will look for an installed version of zmq and compile if needed.

zeromq/pyzmq#205

This fixes #6 and maybe #7.

I ran the zerorpc's test suite with pyzmq 2.2.0.1 in a fresh virtualenv and it passed (but don't take my word).

zerorpc-node introduces a new introspection method that removes some of the legacy issues, called _zpc_inspect. The python version should support this as well.

I have a situation whereby my server also needs to make calls as a client on another server. One half of the design is a Node.js child, and the other is a bit of a gateway written in Python. The Python side receives external events and needs to asynchronously issue ZeroRPC calls to the Node.js child.

At this point, the call does execute (the messages all pass, etc.) but a short time later several exceptions occur indicating there is a problem with gevent.

I've tried spawning the client using gevent, but the exceptions still occur. Ultimately, these exceptions crash ZeroRPC which takes down the Node.js child which then forces the Node.js server to shutdown.

I've posted the additional details to stack overflow as well:

http://stackoverflow.com/questions/21192078/bidirectional-zerorpc-in-python-client-invoke-causes-assertionerror

Is there an example available somewhere that details how to establish a two-way connection?

Hello,

I think this library is amazing and am very impressed with the idea behind zerorpc. The problem is I'm trying to understand how to properly implement the pub/sub pattern with zerorpc.

I know there are tests that illustrate this, but I can't figure out how to take the test and separate the logic into a real world example like the examples you have on http://zerorpc.dotcloud.com page.

I would submit a patch of documentation if I understand the way it works myself but so far I can't figure it out.

Also, how can I safely release or close a a zerorpc connection? If I Ctrl-Z the app while it's running the address doesn't get release and I get Address In Use errors from ZMQ. At this point, I can't restart the process. I see how you guys deal with it in the test documents by creating a random ipc channel each time the test is run but how is the deal with, when tcp addresses are dedicated and you need to restart or recover from a crash.

Test case for Reproduction:

import zerorpc

import time
from gevent import monkey; monkey.patch_all()                                                                                                                                                           

class Foo(object):

    def wait(self):
        print "Start waitin"
        time.sleep(15)
        print "End waitin"
        return "derp"

s = zerorpc.Server(Foo())
s.bind("tcp://0.0.0.0:3333")
s.run()

and the client:

import zerorpc

s = zerorpc.Client("tcp://0.0.0.0:3333", timeout=3000)                                                                                                                                                      
print s.wait()

The result for me on the server is:

/!\ gevent_zeromq BUG /!\ catching after missing event /!
zerorpc.ChannelMultiplexer, unable to route event: _zpc_hb {'response_to': 'e09d15b1-6599-495b-a54d-031e6e7b4039', 'zmqid': ['\x00\xa9\x15J\x94\x8b|O\xf1\xae\x17\xb3M\x82\x8b\xee\x17'], 'message_id': 'e09d15b4-6599-495b-a54d-031e6e7b4039', 'v': 3} [...]

while on the client I get "derp" fine. Without the monkey patch, predictably, the connection is lost due to lack of heartbeat because sleep blocks.

This is a simplification of a larger bug I've been having when dealing with a bunch of workers that all call subprocess to spawn an external executable and do something that takes a while. (With the newest gevent from trunk, there is a subprocess monkey patch).

Any ideas? Zerorpc 0.2.1, gevent 1.0.

I noted in your PyCon talk that you have looked at auto discovery functionality.

When I looked for something similar to use in the meantime I found this, which might be of interest to you:

http://code.google.com/p/minusconf/

Seems to be a very minimalistic and robust approach without any dependencies.

We had a few zeroservice crashing on this recently:

Traceback (most recent call last):
  File "/home/dotcloud/env/lib/python2.6/site-packages/gevent/greenlet.py", line 390, in run
    result = self._run(*self.args, **self.kwargs)
  File "/home/dotcloud/env/lib/python2.6/site-packages/gevent_zerorpc/zerorpc/channel.py", line 76, in _channel_dispatcher
    event = self._events.recv()
  File "/home/dotcloud/env/lib/python2.6/site-packages/gevent_zerorpc/zerorpc/events.py", line 252, in recv
    event = Event.unpack(blob)
  File "/home/dotcloud/env/lib/python2.6/site-packages/gevent_zerorpc/zerorpc/events.py", line 147, in unpack
    (header, name, args) = unpacker.unpack()
TypeError: 'int' object is not iterable
<Greenlet at 0x136b690: <bound method ChannelMultiplexer._channel_dispatcher of <gevent_zerorpc.zerorpc.channel.ChannelMultiplexer object at 0x1477810>>> failed with TypeError

I don't know how it exactly happened, but ZeroRPC shouldn't crash on bad input here.

This happens from time to time with 40+ clients that all have a lot of network and cpu load. I have no idea it this causes the request to fail, but I'm sure it crashes the current (heartbeat?) greenlet, which is never a good thing.

Traceback (most recent call last):
  File "/madcrew/applications/software/virtualenvs/otis_v0_1_0-linux/lib/python2.7/site-packages/gevent/greenl
et.py", line 328, in run
    result = self._run(*self.args, **self.kwargs)
  File "/madcrew/applications/software/virtualenvs/otis_v0_1_0-linux/lib/python2.7/site-packages/zerorpc/heart
beat.py", line 79, in _heartbeat
    self._channel.emit('_zpc_hb', (0,))  # 0 -> compat with protocol v2
AttributeError: 'NoneType' object has no attribute 'emit'
<Greenlet at 0x70a7370: <bound method HeartBeatOnChannel._heartbeat of <zerorpc.heartbeat.HeartBeatOnChannel o
bject at 0x88c4c90>>> failed with AttributeError

We also get a lot of these, but I suspect we are to aggressive calling combined with a high network load, which makes it acceptable:

/!\ gevent_zeromq BUG /!\ catching up after missing event (RECV) /!\

I was wondering what this bit of code is for at the end of the recv function in gevent_zmq.py:

        self._readable.clear()
        while not self._readable.wait(timeout=0.5):
            events = self.getsockopt(_zmq.EVENTS)
            if bool(events & _zmq.POLLIN):
                print "/!\\ gevent_zeromq BUG /!\\ " \
                    "catching after missing event /!\\"
                break

In testing with a couple of sample apps, I hit the print statement once (not reproducible so far...) I was testing making and breaking client tcp connections from a server. I'm running on Windows Vista.

What does "catching after missing event" mean? What conditions/assumptions are being violated here?

Thanks,
Jeff

I'm building a service that (crons) that could take hours/day before activate a client. So for the majority of the time, the client(s) will be waiting for a job.

How can I keep alive the client, and still get new task for a server when ready?

Think in something like pubnub.com

We've seen a few more cases in which ZMQError(EINTR) is raised and not handled in zerorpc.

Unfortunately, these show up in our production systems and not in tests, so I don't have anything else but Tracebacks.

I'll comment the Tracebacks and send a PR with the changes that deal with these exceptions

Using Node.js's spawn(), I create a ZeroRPC server as an IPC. Later when the client times out, my Node sends kill() (SIGTERM) to the child process which is captured by the python script. From the handler, I call stop() on the server which promptly crashes with exceptions. The same set of exceptions occurs if calling close() on the server. What's the appropriate way to ensure a clean shutdown and closing of a python ZeroRPC server?

See the following link for more details: http://stackoverflow.com/questions/21140384/zerorpc-python-server-exceptions-when-attempting-to-stop-or-close

This look very neat, but for develop client the requirements of install 0mq and the rest become complicated. A http/rest interface could be the thing that open this to the world.

Trying out zerorpc python first time gave me the following error

$ zerorpc --server --bind tcp://127.0.0.1:1234 timeTraceback (most recent call last):
  File "/Users/zzz/virtualenvs/zrpc/bin/zerorpc", line 34, in 
    import zerorpc
  File "/Users/zzz/virtualenvs/zrpc/lib/python2.7/site-packages/zerorpc/__init__.py", line 27, in 
    from .context import *
  File "/Users/zzz/virtualenvs/zrpc/lib/python2.7/site-packages/zerorpc/context.py", line 30, in 
    import gevent_zmq as zmq
  File "/Users/zzz/virtualenvs/zrpc/lib/python2.7/site-packages/zerorpc/gevent_zmq.py", line 33, in 
    import gevent.event
  File "/Users/zzz/virtualenvs/zrpc/lib/python2.7/site-packages/gevent/__init__.py", line 41, in 
    from gevent import core
ImportError: dlopen(/Users/zzz/virtualenvs/zrpc/lib/python2.7/site-packages/gevent/core.so, 2): Symbol not found: _event_global_current_base_
  Referenced from: /Users/zzz/virtualenvs/zrpc/lib/python2.7/site-packages/gevent/core.so
  Expected in: flat namespace
 in /Users/zzz/virtualenvs/zrpc/lib/python2.7/site-packages/gevent/core.so

zerorpc version: 0.2.1
operating system: OS X 10.7.4
python: 2.7

Just created a virtualenv, and easy_install'ed zeropc and tried running the server code from the project's homepage
It doesn't work:

zerorpc-with-zmq-13.0.0 ⮀ 18:17:33 ⮀ guyr-air ⮀ ...tmp/zeorpc-test ⮀ python server.py
Traceback (most recent call last):
  File "server.py", line 7, in <module>
    s = zerorpc.Server(HelloRPC())
  File "/usr/local/virtualenvs/zerorpc-with-zmq-13.0.0/lib/python2.7/site-packages/zerorpc-0.4.1-py2.7.egg/zerorpc/core    .py", line 263, in __init__    
    SocketBase.__init__(self, zmq.XREP, context)
AttributeError: 'module' object has no attribute 'XREP'

This started happening today, after pyzmq uploaded 0.13.0.0

We've been experiencing issues with zerorpc and gevent.subprocess.

Consider the following test:

from unittest import TestCase
from time import time
from gevent import spawn, sleep, event
from gevent.subprocess import Popen
from zerorpc import Server, Client


class Service(object):
    def get_nothing(self):
        pass


class BackgroundJobTestCase(TestCase):
    def setUp(self):
        self.stop_event = event.Event()
        self.background_job = spawn(self._start_job)

    def _start_job(self):
        while not self.stop_event.is_set():
            Popen("sleep 1", shell=True).wait()  # this does not work
            # sleep(1)  # this works

    def tearDown(self):
        self.stop_event.set()
        self.background_job = self.background_job.join()

    def test_rpc_with_background_job_for_longer_periods(self, duration_in_seconds=600):
        server = Server(Service())
        server.bind("tcp://0.0.0.0:7001")
        server._acceptor_task = spawn(server._acceptor)  # do not worry about teardown - the test fails in the middle
        client = Client()
        client.connect("tcp://0.0.0.0:7001")
        start_time = time()
        while abs(time() - start_time) <= duration_in_seconds:
            client.get_nothing()

Running this on CentOS 6.4 fails with:

bin/nosetests tests/long_tests/zerorpc_and_subprocess.py                   13-06-04 19:50
Traceback (most recent call last):
  File "/root/mainline/eggs/gevent-1.0rc2-py2.7-linux-x86_64.egg/gevent/greenlet.py", line 328, in run
    result = self._run(*self.args, **self.kwargs)
  File "/root/mainline/eggs/zerorpc-0.4.1.2-py2.7.egg/zerorpc/events.py", line 63, in _sender
    self._socket.send(parts[-1])
  File "/root/mainline/eggs/zerorpc-0.4.1.2-py2.7.egg/zerorpc/gevent_zmq.py", line 104, in send
    self._on_state_changed()
  File "/root/mainline/eggs/zerorpc-0.4.1.2-py2.7.egg/zerorpc/gevent_zmq.py", line 72, in _on_state_changed
    events = self.getsockopt(_zmq.EVENTS)
  File "socket.pyx", line 388, in zmq.core.socket.Socket.get (zmq/core/socket.c:3713)
  File "checkrc.pxd", line 21, in zmq.core.checkrc._check_rc (zmq/core/socket.c:5859)
ZMQError: Interrupted system call
<Greenlet at 0x284c5f0: <bound method Sender._sender of <zerorpc.events.Sender object at 0x2864a50>>> failed with ZMQError

E
======================================================================
ERROR: test_rpc_with_background_job_for_longer_periods (zerorpc_and_subprocess.BackgroundJobTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/mainline/tests/long_tests/zerorpc_and_subprocess.py", line 37, in test_rpc_with_background_job_for_longer_periods
    client.get_nothing()
  File "/root/mainline/eggs/zerorpc-0.4.1.2-py2.7.egg/zerorpc/core.py", line 256, in <lambda>
    return lambda *args, **kargs: self(method, *args, **kargs)
  File "/root/mainline/eggs/zerorpc-0.4.1.2-py2.7.egg/zerorpc/core.py", line 241, in __call__
    return self._process_response(request_event, bufchan, timeout)
  File "/root/mainline/eggs/zerorpc-0.4.1.2-py2.7.egg/zerorpc/core.py", line 213, in _process_response
    reply_event = bufchan.recv(timeout)
  File "/root/mainline/eggs/zerorpc-0.4.1.2-py2.7.egg/zerorpc/channel.py", line 262, in recv
    event = self._input_queue.get(timeout=timeout)
  File "/root/mainline/eggs/gevent-1.0rc2-py2.7-linux-x86_64.egg/gevent/queue.py", line 200, in get
    result = waiter.get()
  File "/root/mainline/eggs/gevent-1.0rc2-py2.7-linux-x86_64.egg/gevent/hub.py", line 568, in get
    return self.hub.switch()
  File "/root/mainline/eggs/gevent-1.0rc2-py2.7-linux-x86_64.egg/gevent/hub.py", line 331, in switch
    return greenlet.switch(self)
LostRemote: Lost remote after 10s heartbeat

----------------------------------------------------------------------
Ran 1 test in 102.469s

FAILED (errors=1)

with gevent-1.0rc2 and zerorpc-0.4.1

How are you guys deploying these services? I'm wondering about configuration, start/stop/reload and monitoring. Thanks.

Hi,

Is it possible to configure multiple workers (pool) with ZeroRPC and have requests distributed across the pool based on the availability the workers?

Hello, in your example :

/Python server/
s = zerorpc.Server(HelloRPC())
s.bind("tcp://0.0.0.0:4242")
s.run()

Could we have say two classes in the server code ? :

/Idea for python server/
import zerorpc

class HelloRPC(object):
def hello(self, name):
return "Hello, %s" % name

class HelloRPC2(object):
def hello(self, name):
return "Hello, %s" % name

s = zerorpc.Server()
s.bind("tcp://0.0.0.0:4242")
s.run()

and choose dynamically which class to execute thanks to a parameter sent by the client ?

/Python Client/
import zerorpc

c = zerorpc.Client()
c.connect("tcp://127.0.0.1:4242")
print c.HelloRPC2.hello("RPC")

Same question for nodejs..

Thanks !

It will make implementation much easier and lower the brier for contributions.

I noticed that the pypi version has some stray print statements which are removed in the repo code. They are very distracting and clutters up the output in the console. I have located them in zerorpc/core.py:148, but as you have removed in the repo them you probably already know about it. Could you release a new pypi verison without them?

def _acceptor(self):
        try:
            while True:
                print '_acceptor read...'
                initial_event = self._multiplexer.recv()
                print '_acceptor read:', initial_event
                self._task_pool.spawn(self._async_task, initial_event)
        except BaseException as e:
            print '_acceptor', e
            raise

After installing zerorpc on linux mint I get this error when trying to use the test server:

python zerorpc --server tcp://_:1234 time
Traceback (most recent call last):
File "zerorpc", line 33, in
exit(cli.main())
File "/usr/local/lib/python2.7/dist-packages/zerorpc-0.4.3-py2.7.egg/zerorpc/cli.py", line 271, in main
return run_server(args)
File "/usr/local/lib/python2.7/dist-packages/zerorpc-0.4.3-py2.7.egg/zerorpc/cli.py", line 111, in run_server
server = zerorpc.Server(server_obj, heartbeat=args.heartbeat)
File "/usr/local/lib/python2.7/dist-packages/zerorpc-0.4.3-py2.7.egg/zerorpc/core.py", line 263, in init
SocketBase.init(self, zmq.ROUTER, context)
File "/usr/local/lib/python2.7/dist-packages/zerorpc-0.4.3-py2.7.egg/zerorpc/socket.py", line 33, in init
self._context = context or Context.get_instance()
File "/usr/local/lib/python2.7/dist-packages/zerorpc-0.4.3-py2.7.egg/zerorpc/context.py", line 98, in get_instance
Context._instance = Context()
File "/usr/local/lib/python2.7/dist-packages/zerorpc-0.4.3-py2.7.egg/zerorpc/context.py", line 49, in init
self._reset_msgid()
File "/usr/local/lib/python2.7/dist-packages/zerorpc-0.4.3-py2.7.egg/zerorpc/context.py", line 104, in _reset_msgid
self._msg_id_counter_stop = random.randrange(self.msg_id_counter, 2*32)
File "/usr/local/lib/python2.7/dist-packages/zerorpc-0.4.3-py2.7.egg/zerorpc/context.py", line 81, in _msg_id_counter
return self.dict['_msg_id_counter']
KeyError: '_msg_id_counter'

Was wondering why zerorpc uses it's own implementation of Socket() instead of using zmq.green.*

(zerorpc)[last: 0] marca@SCML-MarcA:~/dev/git-repos/zerorpc-python$ python setup.py develop
Traceback (most recent call last):
  File "setup.py", line 25, in <module>
    import zerorpc
  File "/Users/marca/dev/git-repos/zerorpc-python/zerorpc/__init__.py", line 32, in <module>
    from .context import *
  File "/Users/marca/dev/git-repos/zerorpc-python/zerorpc/context.py", line 29, in <module>
    import gevent_zmq as zmq
  File "/Users/marca/dev/git-repos/zerorpc-python/zerorpc/gevent_zmq.py", line 33, in <module>
    import gevent.event
ImportError: No module named gevent.event

The dependencies are there in setup.py, but the problem is that setup.py imports zerorpc which imports stuff like gevent before it gets a chance to install them.

I'll take a stab at fixing this. Expect a pull request soon...

this is the example where the server defines an exception class and
the client calls the exception.

running ubuntu, client blows up with this error:

An error occurred: Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/zerorpc-0.4.4-py2.7.egg/zerorpc/core.py", line 148, in _async_task
functor.pattern.process_call(self._context, bufchan, event, functor)
File "/usr/local/lib/python2.7/dist-packages/zerorpc-0.4.4-py2.7.egg/zerorpc/patterns.py", line 30, in process_call
result = functor(*req_event.args)
File "/usr/local/lib/python2.7/dist-packages/zerorpc-0.4.4-py2.7.egg/zerorpc/decorators.py", line 44, in call
return self._functor(_args, *_kargs)
File "zerorpc_server3.py", line 5, in bad
raise Exception(":P")
Exception: :P

the server reports the error below, but does not blow up:

No handlers could be found for logger "zerorpc.core"

I'm considering to create a Perl implementation of ZeroRPC.

Where can I find documentation on how ZeroMQ and MessagePack is wired together to create ZeroRPC?

Is there a formal protocol definition documented anywhere?

How do you handle encryption and authentication between the endpoints, or is that outside the scope of the protocol?

Hi, I've just started using zerorpc, and got stuck trying to use kwargs.
It seems to me that sending remote procedure parameters in kwargs is not possible, but supposed to be:
in ClientBase there is available kwargs:

def __call__(self, method, *args, **kargs):

but further:

    bufchan = BufferedChannel(hbchan, inqueue_size=kargs.get('slots', 100))
    ...
    bufchan.emit(method, args, xheader)

transmitting kwargs just disappeared.
Why ? Did I miss something ? Are kwargs used for something special ?

Thanks in advance.

as of pyzmq==2.2.0.1 gevent support is included via zmq.green as per zeromq/pyzmq#190

Since you already required >=2.2.0.1 in setup.py it would make sense to remove the gevent_zeromq.py and use zmq.green directly.

Just an observation

Trying to follow along with Jérôme's presentation and with the README.rst -- I can't get it to work with --bind.

$ zerorpc-client --server --bind tcp://0:1234 urllib
...
  File "/Users/marca/dev/git-repos/zerorpc-python/zerorpc/events.py", line 212, in bind
    r.append(self._socket.bind(endpoint_))
  File "socket.pyx", line 390, in zmq.core.socket.Socket.bind (zmq/core/socket.c:3613)
zmq.core.error.ZMQError: Operation not supported by device

$ zerorpc-client --server --bind tcp://0:1234 time
...
  File "/Users/marca/dev/git-repos/zerorpc-python/zerorpc/events.py", line 212, in bind
    r.append(self._socket.bind(endpoint_))
  File "socket.pyx", line 390, in zmq.core.socket.Socket.bind (zmq/core/socket.c:3613)
zmq.core.error.ZMQError: Operation not supported by device

Without --bind, not sure what that means though:

(zerorpc)[last: 0] marca@SCML-MarcA:~/dev/git-repos/zerorpc-python$ zerorpc-client --server tcp://0:1234 time
connecting to "tcp://0:1234"
serving "time"
Assertion failed: rc == 0 (src/zmq_connecter.cpp:48)
Abort trap

(zerorpc)[last: 0] marca@SCML-MarcA:~/dev/git-repos/zerorpc-python$ zerorpc-client --server tcp://localhost:1234 time
connecting to "tcp://localhost:1234"
serving "time"

I'm running into some really strange behavior between a python zerorpc server and a zerorpc-node client - there are no logs or errors to help me diagnose why this happens, but when I use the streaming feature, after the client receives all of the messages, the server will simply stop accepting any new requests. Clients can connect fine and attempt to send messages, but they will always time out until I restart the server.

I had to end up using the streaming feature because there are some potentially (very) long running processes, however this isn't a simple timeout issue, although I am also occasionally encountering the same issue mentioned in #37.

I don't see any obvious starting points to help troubleshoot this other than putting log statements throughout the zerorpc code, but I thought I'd open this issue in case this has been encountered before and there is a solution.

I'm running a very simple server as such:

import zerorpc
import gevent


class TestServer:
    def test(self, sentence):
        gevent.sleep(31)
        return sentence + "done"

s = zerorpc.Server(TestServer())
s.bind("tcp://0.0.0.0:8003")
s.run()

when I connect with a client
oc = zerorpc.Client(connect_to="tcp://127.0.0.1:8003")
oc.test('whats up')

I get a TimeoutExpired exception as I was expecting. However, when I use the following code
oc = zerorpc.Client(connect_to="tcp://127.0.0.1:8003")
oc.test('whats up', async=True).wait()

I get the error "AttributeError: 'str' object has no attribute 'name'" in core.py line 220. This was a bit unexpected and I was wondering whether anyone else is seeing the same?

It seems unnecessary to require argparse for python 2.7 and above as that is included in the standard libraries. Or is there another reason for always having it as a dependency?

(tests.test_reqstream.test_rcp_streaming) fails for me (TypeError: can't serialize xrange(10)) with msgpack-python==0.1.13.