zopefoundation / zeo Goto Github PK

ZODB Client-Server framework

License: Other

Python 97.69% Cython 2.31%

zodb database-server database maintained

zeo's Introduction

ZEO - Single-server client-server database server for ZODB

ZEO is a client-server storage for ZODB for sharing a single storage among many clients. When you use ZEO, a lower-level storage, typically a file storage, is opened in the ZEO server process. Client programs connect to this process using a ZEO ClientStorage. ZEO provides a consistent view of the database to all clients. The ZEO client and server communicate using a custom protocol layered on top of TCP.

Some alternatives to ZEO:

NEO is a distributed-server client-server storage.
RelStorage leverages the RDBMS servers to provide a client-server storage.

The documentation is available on readthedocs.

zeo's People

Contributors

Stargazers

Watchers

Forkers

snordhausen krilivye jean e42s janssen nucypher yimai-io lbzhao28 pyzh hvelarde mamico navytux syslabcom azmeuk mrsipan william3johnson inferno-inc oog-gamro

zeo's Issues

ZEO ClientStorage wait_timeout is ignored on Windows

I was writing unit tests for zodbbrowser when I noticed that under Windows the following code never times out:

from ZEO.ClientStorage import ClientStorage
storage = ClientStorage('/no/such/zeo/socket', wait_timeout=0.001, strorage='1', read_only=True)

Instead it hangs forever (I killed the hung Jenkins job after 19 hours).

The timeout works fine on Linux.

ZConfig 3.2.0 leads to stale socket files on Python 2

A change in ZConfig 3.2.0 leads to socket binding addresses being text on both Python2 and Python3.

This breaks runzeo.py's ZEOServer.clear_socket method (on Python 2) which only removes the socket file if it is the type of a string literal:

def clear_socket(self):
  if isinstance(self.options.address, type("")):
     try:
        os.unlink(...)

Under Python 2, the string literal is a bytes object, not a text object.

Now, this wasn't a documented ZConfig change and it wasn't even exactly intended (it was the result of consistency improvements between the behaviour of Python 2 and Python 3, plus bug fixes). Still, because this project already has a dependency on six, it seems to make sense to use six.string_types there instead of type(""), which would solve the issue.

Long connection time

I am experiencing an issue where it takes up to two minutes for the clients to set up a first connection (see below). Is is related to the time re-building the cache ? Note that my client and server are running on the same host.

04.01.16 07:55:06 [+0100] INFO:ZEO.ClientStorage:/var/run/zeo.sock ClientStorage (pid=60) created RW/normal for storage: '1'
04.01.16 07:55:06 [+0100] INFO:ZEO.cache:created temporary cache file '<fdopen>'
04.01.16 07:55:07 [+0100] WARNING:ZEO.zrpc:(60) CW: error connecting to /var/run/zeo.sock: ENOENT
04.01.16 07:55:08 [+0100] WARNING:ZEO.zrpc:(60) CW: error connecting to /var/run/zeo.sock: ENOENT
04.01.16 07:55:10 [+0100] WARNING:ZEO.zrpc:(60) CW: error connecting to /var/run/zeo.sock: ENOENT
04.01.16 07:55:14 [+0100] WARNING:ZEO.zrpc:(60) CW: error connecting to /var/run/zeo.sock: ENOENT
04.01.16 07:55:22 [+0100] WARNING:ZEO.zrpc:(60) CW: error connecting to /var/run/zeo.sock: ENOENT
04.01.16 07:55:38 [+0100] WARNING:ZEO.zrpc:(60) CW: error connecting to /var/run/zeo.sock: ENOENT
04.01.16 07:56:08 [+0100] INFO:ZEO.ClientStorage:/var/run/zeo.sock Testing connection <ManagedClientConnection /var/run/zeo.sock>
04.01.16 07:56:08 [+0100] INFO:ZEO.zrpc.Connection('C'):(/var/run/zeo.sock) received handshake 'Z3101'
04.01.16 07:56:08 [+0100] INFO:ZEO.ClientStorage:/var/run/zeo.sock Server authentication protocol None
04.01.16 07:56:08 [+0100] INFO:ZEO.ClientStorage:/var/run/zeo.sock Connected to storage: /var/run/zeo.sock
04.01.16 07:56:08 [+0100] INFO:ZEO.ClientStorage:/var/run/zeo.sock No verification necessary -- empty cache

Officially support Python 3.7

Attempts at Python2-compatible pickles cause unpicklable objects

Under Py3k, ZODB._compat uses 'zodbpickle', but restricts the maximum protocol level to 2, and passes 'bytes_as_strings=True' to the pickling / unpickling methods. This combination causes some values to be unpicklable. E.g., see this gist:

https://gist.github.com/tseaver/5726422

Against the ZODB trunk, the second stanza fails, because the first has written unpicklable data to the 'tree' object.

Add coverage tests

It would be nice to test the coverage of this project and to kept track of it using https://coveralls.io.

bug: ZEO-Client on andorid

Maybe you are interested in the fact, that ZEO-client works on android platform (using buildozer as android build-platform) as well, but only with a bug-fix which was also discussed for windows clients years before.
I wonder why this bug-fix was not implemented up to now.
change zrpc/client.py line 456 to:
in socket.getaddrinfo(host or 'localhost', port , 0, socket.SOCK_STREAM):

best regards,
Oliver

Return lastTransaction from register

To simplify registration.

auth_digest.get_random_bytes() wastes CPU time

The get_random_bytes function in src/ZEO/auth/auth_digest.py wastes lots of CPU time when it reads from /dev/urandom: Internally, the f.read(n) always reads at least 4kB of precious random data, regardless of how many random bytes you actually want.

This is a result of an optimization of libc or Python that is useful for ordinary files, where it reduces the number of syscalls you have to make to read a file. The behavior can be verified using the "strace" tool, which will show a result similar to this:

stat("/dev/urandom", {...}) = 0
open("/dev/urandom", O_RDONLY) = 3
fstat(3, {...}) = 0
fstat(3, {...}) = 0
ioctl(3, ...) = -1 EINVAL (Invalid argument)
read(3, "\314kp\242Z\320\344`\333\253\352\375\241\6b\310'\351\322\236e\375\203\21\322u4\376{\360V("..., 4096) = 4096
close(3) = 0

The 6th line shows that 4096 bytes are read. Rewriting the function as

def get_random_bytes(n=8):
    try:
        b = os.urandom(n)
    except NotImplementedError:
        L = [chr(random.randint(0, 255)) for i in range(n)]
        b = "".join(L)
    return b

will result in an strace that only shows

open("/dev/urandom", O_RDONLY) = 3
read(3, "\276\3343s\23_}\270", 8) = 8
close(3) = 0

which means the kernel has to produce only 8 bytes of pseudo-random data, not 4096. Additionally, a lot less syscalls are made. In my tests (64 bit Linux), the new function was ~48 times faster for generating 8 random bytes in both Python2 and Python3.

Server keeps disconnecting(?)

Hi,

I'm a bit puzzled, I am running a Plone cluster (2 Zope clients, 1 ZEO master and 1 ZEO replica, with ZRS) and exactly every 10 seconds I get this messages:

2019-02-27T11:16:20 INFO ZEO.asyncio.base Connected server protocol ('SERVER_IP', 9000)
------
2019-02-27T11:16:20 INFO ZEO.StorageServer (unconnected) disconnected

The SERVER_IP is me adding debug statements on ZEO.asyncio.base to know which server is actually the one logging this.

I do get these messages even if all servers are down but the ZEO master (i.e. the 2 Zope clients and the ZEO replica).

As it is an INFO message it should be harmless, though, filling log files every 10 seconds the whole day long is probably not much of an INFO 😅

I'm using Python 2.7 if that matters.

If I add

import traceback
traceback.print_stack()

just before the second message (from ZEO.StorageServer module), I get:

  File "/srv/s-derfreitag/deployment/work/zeo/bin/zeo", line 319, in <module>
    + sys.argv[1:]))
  File "/srv/s-derfreitag/deployment/work/zeo/eggs/plone.recipe.zeoserver-2.0.0-py2.7.egg/plone/recipe/zeoserver/ctl.py", line 35, in main
    runzeo.main(args)
  File "/srv/s-derfreitag/deployment/work/zeo/eggs/ZEO-5.1.1-py2.7.egg/ZEO/runzeo.py", line 390, in main
    s.main()
  File "/srv/s-derfreitag/deployment/work/zeo/eggs/ZEO-5.1.1-py2.7.egg/ZEO/runzeo.py", line 154, in main
    self.loop_forever()
  File "/srv/s-derfreitag/deployment/work/zeo/eggs/ZEO-5.1.1-py2.7.egg/ZEO/runzeo.py", line 255, in loop_forever
    self.server.loop()
  File "/srv/s-derfreitag/deployment/work/zeo/eggs/ZEO-5.1.1-py2.7.egg/ZEO/asyncio/server.py", line 259, in loop
    self.event_loop.run_forever()
  File "/srv/s-derfreitag/deployment/work/zeo/eggs/trollius-2.1-py2.7.egg/trollius/base_events.py", line 309, in run_forever
    self._run_once()
  File "/srv/s-derfreitag/deployment/work/zeo/eggs/trollius-2.1-py2.7.egg/trollius/base_events.py", line 1217, in _run_once
    handle._run()
  File "/srv/s-derfreitag/deployment/work/zeo/eggs/trollius-2.1-py2.7.egg/trollius/events.py", line 136, in _run
    self._callback(*self._args)
  File "/srv/s-derfreitag/deployment/work/zeo/eggs/trollius-2.1-py2.7.egg/trollius/selector_events.py", line 624, in _call_connection_lost
    self._protocol.connection_lost(exc)
  File "/srv/s-derfreitag/deployment/work/zeo/eggs/ZEO-5.1.1-py2.7.egg/ZEO/asyncio/server.py", line 57, in connection_lost
    self.zeo_storage.notify_disconnected()
  File "/srv/s-derfreitag/deployment/work/zeo/eggs/ZEO-5.1.1-py2.7.egg/ZEO/StorageServer.py", line 119, in notify_disconnected
    traceback.print_stack()

Where I see a self._protocol.connection_lost(exc), which leads me to think that somehow the connection is being cut 🤔

Any ideas/tips on how to keep debugging this? TIA!

Confusing installation failures with PyPy 2.5.0 or Python 2.7.8

Trying to install ZEO 5.0 with the PyPy that's on Travis currently fails in a very confusing way:

7.38s$ pip install -U -e ".[test]"
Obtaining file:///home/travis/build/zodb/relstorage
Collecting perfmetrics (from RelStorage==2.0.0b7.dev0)
/home/travis/virtualenv/pypy-2.5.0/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
/home/travis/virtualenv/pypy-2.5.0/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Downloading perfmetrics-2.0.tar.gz
Collecting ZODB>=4.4.2 (from RelStorage==2.0.0b7.dev0)
  Downloading ZODB-5.0.0.tar.gz (467kB)
    100% |████████████████████████████████| 471kB 2.2MB/s 
Collecting ZEO>=4.2.0 (from RelStorage==2.0.0b7.dev0)
  Downloading ZEO-5.0.1.tar.gz (328kB)
    100% |████████████████████████████████| 337kB 3.0MB/s 
No files/directories in /tmp/pip-build-KcLvJI/ZEO/pip-egg-info (from PKG-INFO)

Locally I tracked it down to this command failing:

$ python setup.py egg_info --egg-base pip-egg-info
This version of ZEO requires Python 2.7.9 or higher

(PyPy 2.5.0 claims to be Python 2.7.8.)

pip naturally buries this output by default. What's worse, though is that, when the script exits like that, it exits with a successful exit code:

$ python setup.py egg_info --egg-base pip-egg-info
This version of ZEO requires Python 2.7.9 or higher
$ echo $?
0

(I'm guessing that allowing Python 2.7.8 is not going to happen because SSL. So I'll have to pin versions back just on Travis, just for PyPy.)

However, if we can make the exit code a failure code then pip will at least bail out early with a reasonable error message (I edited the script to sys.exit(1) instead of 0):

$ pip install  ZEO-5.0.1.tar.gz
Processing ./ZEO-5.0.1.tar.gz
    Complete output from command python setup.py egg_info:
    This version of ZEO requires Python 2.7.9 or higher

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /var/folders/y5/x7pvzk651c3dqkllbxd1jd280000gn/T/pip-uraGDb-build/

Can't connect to ZEO server on Windows

This is a duplicate of the issue found here:
https://bugs.launchpad.net/zodb/+bug/1004513

The solution provided by Lubomir Kostov (lkostov) works, and I have tried the fix as well. I can resubmit a PR if needed.

conflict resolution logs coming continuously in zeo.log

I am getting following logs in zeo.log again and again :

2017-06-09T15:31:46 ERROR ZODB.ConflictResolution Unexpected error while trying to resolve conflict on <type 'BTrees.OOBTree.OOBucket'>
Traceback (most recent call last):
File "C:\batonSites\Python27\lib\site-packages\zodb-4.4.3-py2.7.egg\ZODB\ConflictResolution.py", line 285, in tryToResolveConflict
resolved = resolve(old, committed, newstate)
File "C:\batonSites\Python27\lib\site-packages\zodb-4.4.3-py2.7.egg\ZODB\ConflictResolution.py", line 178, in _cmp_
"can't reliably compare against different "
ValueError: can't reliably compare against different PersistentReferences

What could be the reason of getting this type of error at ZEO end ?

Don't transmit error instances for protocol 5

Error handling is awkward in a number of ways:

Detecting errors involves sniffing results in ways that seem like bug magnets.
We're required to pickle error objects, with the security issues that entails.
- We shouldn't need to pickle objects at the protocol level.
- We might want to use alternate serialization formats that don't support object serialization.

Instead, we'll send dotted error class names and arguments and we'll use the async flag as an error flag in replies.

Pickling in marshal.py is not compatible with PyPy 5.0

Tried to run ZEO client/server with PyPy 5.0, getting the following error:

Traceback (most recent call last):
  File "/home/michwill/Projects/zerodb-server/.venvpypy/site-packages/ZEO/zrpc/client.py", line 600, in test_connection
    self.preferred = self.client.testConnection(self.conn)
  File "/home/michwill/Projects/zerodb-server/.venvpypy/site-packages/ZEO/ClientStorage.py", line 575, in testConnection
    auth = stub.getAuthProtocol()
  File "/home/michwill/Projects/zerodb-server/.venvpypy/site-packages/ZEO/ServerStub.py", line 81, in getAuthProtocol
    return self.rpc.call('getAuthProtocol')
  File "/home/michwill/Projects/zerodb-server/.venvpypy/site-packages/ZEO/zrpc/connection.py", line 775, in call
    msgid = self.send_call(method, args)
  File "/home/michwill/Projects/zerodb-server/.venvpypy/site-packages/ZEO/zrpc/connection.py", line 552, in send_call
    buf = self.encode(msgid, async, method, args)
  File "/home/michwill/Projects/zerodb-server/.venvpypy/site-packages/ZEO/zrpc/marshal.py", line 44, in encode
    return pickler.dump(args, 1)
TypeError: dump() takes exactly 2 arguments (3 given)

Can't we use zodbpickle here?

zeoctl is untested and, of course broken.

Why is it even a thing?

Create a release with current changes

Can I please have a new ZEO release either as 5.2.1 or 5.3.0?

Can a persistent cache file be shared by various instances?

I was trying to set only one persistent cache file among 2 instances but I got a lot of LogError exceptions when I tried to start the second one.

I just want to know if this is possible or not:

2017-08-30T19:28:50 ERROR zc.lockfile Error locking file /tmp/zeoclient-1.zec.lock; pid=10889
Traceback (most recent call last):
  File "/opt/plone/buildout/eggs/zc.lockfile-1.0.2-py2.7.egg/zc/lockfile/__init__.py", line 84, in __init__
    _lock_file(fp)
  File "/opt/plone/buildout/eggs/zc.lockfile-1.0.2-py2.7.egg/zc/lockfile/__init__.py", line 59, in _lock_file
    raise LockError("Couldn't lock %r" % file.name)
LockError: Couldn't lock '/tmp/zeoclient-1.zec.lock'

First read not consistent after write from other thread

I have two threads connecting to a ZODB via ZEO.
Thread A is reading data and thread B is writing data periodically.
The issue I am facing is that first read from A after a write from B is not consistent. Only after that first read does A see the updated objects. I guess this is somehow related to caching but I am not quite sure.
Can you provide me with some guidance ?

Update to msgpack >= 0.6

In #115 the dependencies have been updated from the deprecated msgpack-python to msgpack < 0.6 as there are test failures with newer versions.

It would be good to fix the test failures to be able to use the most recent version.

runzeo error (Plone 5/Python 2.7): Attempted relative import in non-package

In order to not run into the problems of #84 I tried to use runzeo with supervisor.

runzeo fails to start with:

Traceback (most recent call last):
  File "/home/jensens/.buildout/shared-eggs/ZEO-5.1.0-py2.7.egg/ZEO/runzeo.py", line 401, in <module>
    main()
  File "/home/jensens/.buildout/shared-eggs/ZEO-5.1.0-py2.7.egg/ZEO/runzeo.py", line 392, in main
    s.main()
  File "/home/jensens/.buildout/shared-eggs/ZEO-5.1.0-py2.7.egg/ZEO/runzeo.py", line 155, in main
    self.create_server()
  File "/home/jensens/.buildout/shared-eggs/ZEO-5.1.0-py2.7.egg/ZEO/runzeo.py", line 251, in create_server
    self.server = create_server(self.storages, self.options)
  File "/home/jensens/.buildout/shared-eggs/ZEO-5.1.0-py2.7.egg/ZEO/runzeo.py", line 345, in create_server
    from .StorageServer import StorageServer
ValueError: Attempted relative import in non-package

I added a configuration to the Plone coredev buildout here:
https://github.com/plone/buildout.coredev/blob/5.1/experimental/zeoserver.cfg
and run it with ./bin/buildout -c experimental/zeoserver.cfg
The generated script ./bin/zeoserver has two modes, one uses zeoctl and the other, if in enviroment SUPERVISOR_ENABLED=1 is set, runzeo is used.

After changing the relative import to an absolute one this specific problem is gone and runzeo or SUPERVISOR_ENABLED=1 ./bin/zeoserver works (but #84 is still an issue).

fshelper.checkSecure deprecation warning

When setting a blob directory, the following warning appears:

...ZEO/ClientStorage.py:252: DeprecationWarning: checkSecure is deprecated. Permissions are no longer set by ZODB
    self.fshelper.checkSecure()

My company has thousands of unit tests, and a lot of them are generating this warning in test reports. This makes other warnings hard to read.

Can not connect to ZEO server for packing

Hi,

I'm running Plone and with the latest upgrade to 5.1b4 ZEO 5.1.0 is used which depends on trollius as a Python 2.7 backport of asyncio. Since that I get the following errors when trying to pack the ZODB:

plone.recipe.zeoserver.pack ERROR Could not connect to zeoserver. Please make sure it is running.
trollius ERROR Exception in callback done_connecting(<Task ca....py:562>>) at /opt/plone/default/5.1b4/buildout-cache/eggs/ZEO-5.1.0-py2.7.egg/ZEO/asyncio/client.py:127
handle: <Handle done_connecting(<Task ca....py:562>>) at /opt/plone/default/5.1b4/buildout-cache/eggs/ZEO-5.1.0-py2.7.egg/ZEO/asyncio/client.py:127>
Traceback (most recent call last):
  File "/opt/plone/default/5.1b4/buildout-cache/eggs/trollius-2.1-py2.7.egg/trollius/events.py", line 136, in _run
    self._callback(*self._args)
  File "/opt/plone/default/5.1b4/buildout-cache/eggs/ZEO-5.1.0-py2.7.egg/ZEO/asyncio/client.py", line 129, in done_connecting
    if future.exception() is not None:
  File "/opt/plone/default/5.1b4/buildout-cache/eggs/trollius-2.1-py2.7.egg/trollius/futures.py", line 299, in exception
    raise CancelledError
CancelledError

plone.recipe.zeoserver.pack ERROR Could not connect to zeoserver. Please make sure it is running.
concurrent.futures ERROR exception calling callback for <Future at 0x7f9f3064f890 state=finished returned list>
Traceback (most recent call last):
  File "/opt/plone/default/5.1b4/buildout-cache/eggs/futures-3.1.1-py2.7.egg/concurrent/futures/_base.py", line 301, in _invoke_callbacks
    callback(self)
  File "/opt/plone/default/5.1b4/buildout-cache/eggs/trollius-2.1-py2.7.egg/trollius/futures.py", line 449, in <lambda>
    new_future._copy_state, future))
  File "/opt/plone/default/5.1b4/buildout-cache/eggs/trollius-2.1-py2.7.egg/trollius/base_events.py", line 496, in call_soon_threadsafe
    handle = self._call_soon(callback, args)
  File "/opt/plone/default/5.1b4/buildout-cache/eggs/trollius-2.1-py2.7.egg/trollius/base_events.py", line 470, in _call_soon
    self._check_closed()
  File "/opt/plone/default/5.1b4/buildout-cache/eggs/trollius-2.1-py2.7.egg/trollius/base_events.py", line 297, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

I'm using buildout to configure my setup and this is the zeoserver part:

###############################################################################
# ZEO-Server
# -----------------------------------------------------------------------------
# Use this section to install and configure a Zope Enterprise Objects server.
# For options see http://pypi.python.org/pypi/plone.recipe.zeoserver
[zeoserver]
recipe = plone.recipe.zeoserver
zeo-address = 127.0.0.1:8100

# If we try to start as root, Zope will switch to the user below.
effective-user = ${buildout:effective-user}

environment-vars = ${buildout:environment-vars}

# Set storage.
var = ${buildout:var-dir}
blob-storage = ${buildout:var-dir}/blobstorage

# Put the log, pid and socket files in var/zeoserver.
zeo-log     = ${buildout:var-dir}/zeoserver/zeoserver.log
pid-file    = ${buildout:var-dir}/zeoserver/zeoserver.pid
socket-name = ${buildout:var-dir}/zeoserver/zeo.zdsock

The generated script looks like this:

#!/opt/plone/default/Python-2.7/bin/python2.7

import sys
sys.path[0:0] = [
  '/opt/plone/default/5.1b4/buildout-cache/eggs/plone.recipe.zeoserver-1.4.0-py2.7.egg',
  '/opt/plone/default/5.1b4/buildout-cache/eggs/ZopeUndo-4.2-py2.7.egg',
  '/opt/plone/default/5.1b4/buildout-cache/eggs/zope.mkzeoinstance-4.1-py2.7.egg',
  '/opt/plone/default/5.1b4/buildout-cache/eggs/ZODB3-3.11.0-py2.7.egg',
  '/opt/plone/default/5.1b4/buildout-cache/eggs/zc.recipe.egg-2.0.3-py2.7.egg',
  '/opt/plone/default/Python-2.7/lib/python2.7/site-packages',
  '/opt/plone/default/5.1b4/buildout-cache/eggs/ZEO-5.1.0-py2.7.egg',
  '/opt/plone/default/5.1b4/buildout-cache/eggs/ZODB-5.2.4-py2.7.egg',
  '/opt/plone/default/5.1b4/buildout-cache/eggs/zdaemon-4.2.0-py2.7.egg',
  '/opt/plone/default/5.1b4/buildout-cache/eggs/transaction-2.1.2-py2.7.egg',
  '/opt/plone/default/5.1b4/buildout-cache/eggs/BTrees-4.4.1-py2.7-linux-x86_64.egg',
  '/opt/plone/default/5.1b4/buildout-cache/eggs/persistent-4.2.4.2-py2.7-linux-x86_64.egg',
  '/opt/plone/default/5.1b4/buildout-cache/eggs/trollius-2.1-py2.7.egg',
  '/opt/plone/default/5.1b4/buildout-cache/eggs/futures-3.1.1-py2.7.egg',
  '/opt/plone/default/5.1b4/buildout-cache/eggs/zope.interface-4.4.0-py2.7-linux-x86_64.egg',
  '/opt/plone/default/5.1b4/buildout-cache/eggs/ZConfig-3.1.0-py2.7.egg',
  '/opt/plone/default/5.1b4/buildout-cache/eggs/zc.lockfile-1.2.1-py2.7.egg',
  '/opt/plone/default/5.1b4/buildout-cache/eggs/six-1.10.0-py2.7.egg',
  '/opt/plone/default/5.1b4/buildout-cache/eggs/zodbpickle-0.6.0-py2.7-linux-x86_64.egg',
  ]

username = None
blob_dir = "/opt/plone/default/var/blobstorage"
realm = None
storage = "1"
days = "1"
unix = None
address = "127.0.0.1:8100"
host = "127.0.0.1"
password = None
port = "8100"
import getopt; opts = getopt.getopt(sys.argv[1:], 'S:B:D:W1')[0];
opts = dict(opts)
storage = opts.has_key('-S') and opts['-S'] or storage
blob_dir = opts.has_key('-B') and opts['-B'] or blob_dir
days = opts.has_key('-D') and opts['-D'] or days


import plone.recipe.zeoserver.pack

if __name__ == '__main__':
    sys.exit(plone.recipe.zeoserver.pack.main(host, port, unix, days, username, password, realm, blob_dir, storage))

Any hints what could be wrong?

Memory leak on ZEO server

On some production sites we are using ZEO on a ZRS configuration and I recently started spotting very high memory consumption on the zeoserver process (as high as 1.6GB on one server I just had to restart).

We have currently 2 sites with this issue:

site C: 1 database; 17,277,281 objects, 2 ZEO servers (one ZRS master and one ZRS slave); each ZEO server has 3 ZEO clients attached to it
site R: 1 database; 18,460,812 objects, 2 ZEO servers (one ZRS master and one ZRS slave); each ZEO server has 2 ZEO clients attached to it

Memory usage increases gradually with every database pack in around 170MB steps; and the leak happens with and without ZRS.

Some graphs can be seen here: https://community.plone.org/t/very-high-memory-consumption-on-zeo-server-process/4208

Versions used:

plone.recipe.zeoserver = 1.2.9
ZopeUndo = 2.12.0
ZODB3 = 3.10.5
zope.mkzeoinstance = 3.9.5
zc.recipe.egg = 1.3.2
zc.buildout = 2.5.3
zc.zrs = 2.4.4
zope.interface = 3.6.7
zope.event = 3.5.2
zdaemon = 2.0.7
ZConfig = 2.9.3
zc.lockfile = 1.0.2
transaction = 1.1.1
Twisted = 16.6.0
incremental = 16.10.1
constantly = 15.1.0

UnboundLocalError: local variable

We started seeing the error below today. We are running w/ ZODB 5.4.0

Thx

Carlos

2018-03-26T11:27:54 (/Users/cutz/Documents/NextThought/nti.dataserver-buildout/var/zeosocket) disconnected
2018-03-26T11:27:54 Connected server protocol
2018-03-26T11:27:55 received handshake 'Z5'
2018-03-26T11:27:55 can't decode message: '((ccopy_reg\n_reconstructor\n(czodbpickle\nbinary\nc__b...'
2018-03-26T11:27:55 Can't deserialize message
Traceback (most recent call last):
  File "/Users/cutz/Documents/NextThought/nti.dataserver-buildout/eggs/ZEO-5.1.1-py2.7.egg/ZEO/asyncio/server.py", line 89, in message_received
    message_id, async, name, args = self.decode(message)
  File "/Users/cutz/Documents/NextThought/nti.dataserver-buildout/eggs/ZEO-5.1.1-py2.7.egg/ZEO/asyncio/marshal.py", line 114, in pickle_server_decode
    return unpickler.load() # msgid, flags, name, args
  File "/Users/cutz/Documents/NextThought/nti.dataserver-buildout/eggs/ZEO-5.1.1-py2.7.egg/ZEO/asyncio/marshal.py", line 164, in server_find_global
    raise ImportError("import error %s: %s" % (module, msg))
ImportError: import error zodbpickle: 
2018-03-26T11:27:55 data_received 4 0 True
Traceback (most recent call last):
  File "/Users/cutz/Documents/NextThought/nti.dataserver-buildout/eggs/ZEO-5.1.1-py2.7.egg/ZEO/asyncio/base.py", line 128, in data_received
    self.message_received(collected)
  File "/Users/cutz/Documents/NextThought/nti.dataserver-buildout/eggs/ZEO-5.1.1-py2.7.egg/ZEO/asyncio/server.py", line 94, in message_received
    if message_id == -1:
UnboundLocalError: local variable 'message_id' referenced before assignment

Creating a Windows Service for ZEO

A new thread for discussing ZEO service for Windows.

Use extras_require for Python 2 dependencies

This will let us produce universal wheels.

Files are never deleted from blob cache directory?

Hi,

I am using a ZEO client having this configuration:

shared-blob-dir off

Scenario:

I upload a file in Zope/Plone and access it will create both the blob files (on file-system) in both blob storage and in blob cache directories.
I delete the file from Zope/Plone.
File is deleted from the blob storage but not from the blob cache directory.

I also tried with a ZEO client having bellow configuration and got same result:

shared-blob-dir off
cache-size 350MB
blob-cache-size 70000000
blob-cache-size-check 10

Is this a bug or an intended feature? How can I force the deletion of those files from the blob cache directory?

Thx,
Alec

No files/directories in /tmp/pip-build-sG9CHF/ZEO/pip-egg-info

Collecting ZEO
  Downloading ZEO-5.0.1.tar.gz (328kB)
    100% |████████████████████████████████| 337kB 631kB/s 
No files/directories in /tmp/pip-build-sG9CHF/ZEO/pip-egg-info (from PKG-INFO)

Confirmed on two servers.

It seems to be the problem of the python version, which is 2.7.9, but the package requires 2.7.12. My question is - do you really need the 2.7.12?

I don't want to sound rude, but there is a lot of machines on python 2.7.9, which is the Ubuntu LTS 14.04 version.

runzeo problem with opening storages obscured by cleanup code

In:

def main(self):
    self.setup_default_logging()
    self.check_socket()
    self.clear_socket()
    self.make_pidfile()
    try:
        self.open_storages()
        self.setup_signals()
        self.create_server()
        self.loop_forever()
    finally:
        self.server.close()
        self.clear_socket()
        self.remove_pidfile()

If there's a failure in or before create_server, the first statement in the finally block will fail, hiding the original error.

Why msgpack is not enabled by default?

The README file shows the pros for using msgpack with the ZEO server but does not talk about any drawback.

msgpack

Use msgpack to serialize and de-serialize ZEO protocol messages.

An advantage of using msgpack for ZEO communication is that it's a tiny bit faster and a ZEO server can support Python 2 or Python 3 clients (but not both).

msgpack can also be enabled by setting the ZEO_MSGPACK environment to a non-empty string.

Is there any reason that could prevent someone from using msgpack? If it is the case, maybe the README should talk about them. If not, why isn't it enabled by default?

Reconnecting is mean to the object cache

See hysterical discussion here:

https://mail.zope.org/pipermail/zodb-dev/2006-August/010343.html
(ignore the ssl warning -- sorry)

Which led to this change:

4413420

We clear the object cache because we may have objects in the object cache that aren't in the ZEO client cache, which wouldn't be verified as part of cache verification (which we're getting rid of. If we can do quick verification, which involved replaying missed invalidations, then there's no need to clear the object cache. OTOH, if we drop/clear the client cache, then we should clear the object cache as well. We'll need to make sure that notify_connected (new name for notifyConnected) has enough information to know whether to drop the object cache.

Make ClientDisconnected retryable

Goal is that if we're disconnected, we don't return an error unless we remain disconnected for a long time.

To achieve the goal, we may need to did a little deeper. For example, we currently error immediately if we make an async call while disconnected, but maybe we should block. But maybe only certain methods, like tpc_begin...

LockError errors when using blob-cache-size

Hi,

I am using a ZEO client having this configuration:

shared-blob-dir off
cache-size 350MB
blob-cache-size 70000000
blob-cache-size-check 10

The blob-cache-size works fine and the limit is pretty well kept but the Zope/Plone console is full of the bellow errors. The error happens when a previously deleted blob file from blob cache is accessed:

ERROR zc.lockfile Error locking file /var/work/zodbblobcache/check_size.lock; pid=UNKNOWN
Traceback (most recent call last):
  File "/var/work/myPlone/eggs/zc.lockfile-1.0.0-py2.6.egg/zc/lockfile/__init__.py", line 76, in __init__
    _lock_file(fp)
  File "/var/work/myPlone/eggs/zc.lockfile-1.0.0-py2.6.egg/zc/lockfile/__init__.py", line 59, in _lock_file
    raise LockError("Couldn't lock %r" % file.name)
LockError: Couldn't lock '/var/work/zodbblobcache/check_size.lock'

The Zope/Plone UI is not reporting any errors and files are accessible but the logs are full of this LockError errors. Why is this happening and how it can be avoided? Is this an INFO message wrongly marked as an ERROR?

Thx,
Alec

--configuration option not recognized

-C option works but not --configuration option:

$ ./venv/bin/runzeo -C zeo.config  # ok
$ ./venv/bin/runzeo --configuration zeo.config 
Error: option --configuration not recognized
For help, use ./venv/bin/runzeo -h

The documentation says it should be equivalent:

$ ./venv/bin/runzeo -h
Start the ZEO storage server.

Usage: ./venv/bin/runzeo [-C URL] [-a ADDRESS] [-f FILENAME] [-h]

Options:
-C/--configuration URL -- configuration file or URL
[snip]

It's because the options are parsed by the zdaemon.zeoptions.ZDOptions() class, which use --configure as long parameter to get the configuration file. So there is a difference:

runzeo.__doc__ : -C/--configuration
zdoptions.ZDOoptions.__doc__ : -C/--configure

ZDOoptions documentation fits to its own code, line 73:

self.add("configfile", None, "C:", "configure=")

It's not possible to just add

self.add("configfile", None, "C:", "configuration=")

in runzeo.py:ZEOOptionsMixin.add_zeo_options() because the same -C option is still used by ZDOptions.

Perhaps it would be better to change one of the long option of runzeo.py or ZDOptions class to provide the same long parameter name in both packages?
Cons:

breaks the previous compatibility

Pros:

improve the homogeneity, so easier to use
probably does not break to much because no one can use it currently.

I'm ok to make a PR but I prefer to get feedbacks before.

Tested with package ZEO-5.2.1, python 3.7.

mistake in writing

ZEO/src/ZEO/__init__.py

line 43 ra should be raise

Don't save configs generated by the forker. They just end up being annoying.

Test failures on Mac

checkReconnectSwitch (ZEO.tests.testConnection.FileStorageReconnectionTests)
invalidations_while_connecting (ZEO.tests.testConnection)
gracefully_handle_abort_while_storing_many_blobs (ZEO.tests.testZEO)

Select fails with: OSError: [Errno 9] Bad file descriptor

This seems vaguely familiar. :(

ClientStorage._check_blob_cache_size leaks open file

See https://github.com/zopefoundation/ZEO/blob/master/src/ZEO/ClientStorage.py#L1670

Reported by the RelStorage tests on Py3.

Quick Server-Client setup is broken

ZEO 's quick Server-Client setup seems to depend on a module from its tests folder, which depends on something inside ZODB's tests, which requires a module that existed in ZODB 5.1.1, but no longer in 5.3.0.

Traceback (most recent call last):
  File "/home/qqwy/.local/share/virtualenvs/pydash-bxa1s-wp/bin/flask", line 11, in <module>
    sys.exit(main())
  File "/home/qqwy/.local/share/virtualenvs/pydash-bxa1s-wp/lib/python3.6/site-packages/flask/cli.py", line 513, in main
    cli.main(args=args, prog_name=name)
  File "/home/qqwy/.local/share/virtualenvs/pydash-bxa1s-wp/lib/python3.6/site-packages/flask/cli.py", line 380, in main
    return AppGroup.main(self, *args, **kwargs)
  File "/home/qqwy/.local/share/virtualenvs/pydash-bxa1s-wp/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/qqwy/.local/share/virtualenvs/pydash-bxa1s-wp/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/qqwy/.local/share/virtualenvs/pydash-bxa1s-wp/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/qqwy/.local/share/virtualenvs/pydash-bxa1s-wp/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/qqwy/.local/share/virtualenvs/pydash-bxa1s-wp/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/qqwy/.local/share/virtualenvs/pydash-bxa1s-wp/lib/python3.6/site-packages/flask/cli.py", line 256, in decorator
    with __ctx.ensure_object(ScriptInfo).load_app().app_context():
  File "/home/qqwy/.local/share/virtualenvs/pydash-bxa1s-wp/lib/python3.6/site-packages/flask/cli.py", line 237, in load_app
    rv = locate_app(self.app_import_path)
  File "/home/qqwy/.local/share/virtualenvs/pydash-bxa1s-wp/lib/python3.6/site-packages/flask/cli.py", line 90, in locate_app
    __import__(module)
  File "/run/media/qqwy/Serendipity/Programming/RUG/SoftwareEng/PyDashIO/2018-PyDash.io/pydash/pydash.py", line 1, in <module>
    from pydash_web import flask_webapp
  File "/run/media/qqwy/Serendipity/Programming/RUG/SoftwareEng/PyDashIO/2018-PyDash.io/pydash/pydash_web/__init__.py", line 20, in <module>
    from pydash_web import routes  # Needs to be below flask_webapp instantiation to prevent circular dependency
  File "/run/media/qqwy/Serendipity/Programming/RUG/SoftwareEng/PyDashIO/2018-PyDash.io/pydash/pydash_web/routes.py", line 10, in <module>
    import pydash_web.controller as controller
  File "/run/media/qqwy/Serendipity/Programming/RUG/SoftwareEng/PyDashIO/2018-PyDash.io/pydash/pydash_web/controller/__init__.py", line 5, in <module>
    from .login import login
  File "/run/media/qqwy/Serendipity/Programming/RUG/SoftwareEng/PyDashIO/2018-PyDash.io/pydash/pydash_web/controller/login.py", line 8, in <module>
    import pydash_app.user
  File "/run/media/qqwy/Serendipity/Programming/RUG/SoftwareEng/PyDashIO/2018-PyDash.io/pydash/pydash_app/user/__init__.py", line 6, in <module>
    import pydash_app.user.user_repository
  File "/run/media/qqwy/Serendipity/Programming/RUG/SoftwareEng/PyDashIO/2018-PyDash.io/pydash/pydash_app/user/user_repository.py", line 19, in <module>
    from ..impl.database import database_root, MultiIndexedPersistentCollection
  File "/run/media/qqwy/Serendipity/Programming/RUG/SoftwareEng/PyDashIO/2018-PyDash.io/pydash/pydash_app/impl/database.py", line 12, in <module>
    connection = ZEO.server(path='zeo_filestorage.fs')
  File "/home/qqwy/.local/share/virtualenvs/pydash-bxa1s-wp/lib/python3.6/site-packages/ZEO/__init__.py", line 84, in server
    import os, ZEO.tests.forker
  File "/home/qqwy/.local/share/virtualenvs/pydash-bxa1s-wp/lib/python3.6/site-packages/ZEO/tests/forker.py", line 29, in <module>
    import ZODB.tests.util
  File "/home/qqwy/.local/share/virtualenvs/pydash-bxa1s-wp/lib/python3.6/site-packages/ZODB/tests/util.py", line 29, in <module>
    import zope.testing.setupstack
ModuleNotFoundError: No module named 'zope.testing'

`async` is a keyword in Python 3.7

Python 3.7.0b1+ (heads/3.7:7f38637853, Feb  1 2018, 13:09:53) 
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ZEO.ClientStorage
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/py37/lib/python3.7/site-packages/ZEO/ClientStorage.py", line 271
    self._async = self._server.async
                                   ^
SyntaxError: invalid syntax

Reopening: Cannot install PyQt5 under Python 2.7.14

This issue is a reopening of #77.

My version of python in /usr/local/bin is 2.7.14:84471935ed, Sept 16, 2017.

Calling ./python -m pip install python-qt5 produces this error:

No files/directories in /private/var/folders/cm/60_4h2mj23d_70fhqwvtjf7m0000gn/T/pip-build-MyNknQ/python-qt5/pip-egg-info (from PKG-INFO)

Looks like we're dead in the water until this issue is acknowledged and properly addressed.

Pros and cons of client conflict resolution

I see there is a client-conflict-resolution option, off by default. I could not find documentation about the advantages and drawbacks of using this option. I am guessing that:

The server does not need to have the conflict resolution code on its pythonpath, so it may be easier to use than server-side conflict resolution.
It generates more network communication between the client and the server, so it may be slower that server-side conflict resolution.

Is it something more? Maybe it worth writing some words on the README about this?

ImportError: Unsafe global: copy_reg._reconstructor

We are trying ZEO 5.1.2 and we got the error below. We are using ZODB 5.3.0

Carlos

2018-03-27 13:25:17,315 ERROR [ZEO.asyncio.marshal][4804095152:2209][Users zeo client networking thread] can't decode message: '((ccopy_reg\n_reconstructor\n(czodbpickle\nbinary\nc__b...'
2018-03-27 13:25:17,317 ERROR [ZEO.asyncio.base][4804095152:2209][Users zeo client networking thread] data_received 4 0 True
Traceback (most recent call last):
  File "/Users/Birdwell/Projects/nti.dataserver-buildout/eggs/ZEO-5.1.2-py2.7.egg/ZEO/asyncio/base.py", line 128, in data_received
    self.message_received(collected)
  File "/Users/Birdwell/Projects/nti.dataserver-buildout/eggs/ZEO-5.1.2-py2.7.egg/ZEO/asyncio/client.py", line 206, in message_received
    msgid, async, name, args = self.decode(data)
  File "/Users/Birdwell/Projects/nti.dataserver-buildout/eggs/ZEO-5.1.2-py2.7.egg/ZEO/asyncio/marshal.py", line 91, in pickle_decode
    return unpickler.load() # msgid, flags, name, args
  File "/Users/Birdwell/Projects/nti.dataserver-buildout/eggs/ZEO-5.1.2-py2.7.egg/ZEO/asyncio/marshal.py", line 155, in find_global
    raise ImportError("Unsafe global: %s.%s" % (module, name))
ImportError: Unsafe global: copy_reg._reconstructor

StorageServer.py rises a TypeError

There is a argument error in line 758.

File "/usr/local/lib/python3.5/dist-packages/ZEO/StorageServer.py", line 758, in __init__
    self.acceptor = Acceptor(self, addr, ssl, msgpack)
TypeError: __init__() takes 4 positional arguments but 5 were given

Windows: trigger was using "str" instead of "bytes" -> broken in Python 3.2

Patched here: #15

Add an option to get ZEO to really sync.

By making a server round trip.

Make the server listen on several address

I would love to be able to use a configuration file like this:

<zeo>
    address localhost:8132
    address 10.xx.yy.zz:8132
</zeo>

My company would use it to connect from our private network as well as locally. Right now we can use the private network IP to connect, but I think such a feature would not be so hard to do.

What do you think?

Use python_requires keyword for declarative version checking

Instead of imperatively doing so in setup.py. E.g., pip. This will let PyPI know what python versions we run on, and will let pip and other tools that support that metadata (maybe that's baked into setuptools itself? I'm not sure) choose the appropriate version to install. This is also necessary to be able to ship wheels (because setup.py doesn't run).

Need tests for failure (disconnect and error response) calling get_info during connect on the asyncio branch

I'm pretty sure it isn't handled correctly now and the place where it's called is delicate.