Comments (3)
tseaver
commented on September 21, 2024
The preceding log entry is for a "Can't decode message" error: the 'message_receivedhandler should be returning in theexcept:` clause.
You'll still have the problem that the server closes when the decode fails, though.
from zeo.
jamadden
commented on September 21, 2024
It looks like the decoding fails because zodbpickle is not importable:
File "/Users/cutz/Documents/NextThought/nti.dataserver-buildout/eggs/ZEO-5.1.1-py2.7.egg/ZEO/asyncio/marshal.py", line 164, in server_find_global
raise ImportError("import error %s: %s" % (module, msg))
ImportError: import error zodbpickle:
ZEO unconditionally depends on ZODB, which unconditionally depends on zodbpickle, so that seems fishy. This deployment is via buildout, so it's possible some part definition is not quite correct. I've asked some followup questions about that.
from zeo.
jamadden
commented on September 21, 2024
I'd like to reopen this issue for two reasons:
- All the tests for Python 2.7 using pickle failed after #108 was merged. There are a bunch of
ClientDisconnected: ('tpc_vote', 'on a disconnected transaction’)in the output, which is what we're seeing on our clients. Note that the 2.7 test using the msgpack protocol passed. - @cutz followed the
ImportErrordown, and it appears to be the case thatserver_find_globalsimply won't allowzodbpickle.binaryto be loaded, directly raising theImportErrorwe see above.
So it looks like an additional needed fix is to modify server_find_global to allow zodbpickle as a module name.
Thoughts?
Attaching the analysis from @cutz below:
I think the import error being logged isn't actually an error importing zodbpickle. I'm breaking in ZEO at the point that error is raised:
/Users/cutz/Documents/NextThought/nti.dataserver-buildout/eggs/ZEO-5.1.1-py2.7.egg/ZEO/asyncio/marshal.py(114)pickle_server_decode()
is trying to decode the msg:
'((ccopy_reg\n_reconstructor\n(czodbpickle\nbinary\nc__builtin__\nstr\nU\x08\x00\x00\x00\x00\x00\x03\x1b\x9ctRU\x08\x03\xc6p\x12\xfc$OgtI00\nU\nloadBefore(ccopy_reg\n_reconstructor\n(czodbpickle\nbinary\nc__builtin__\nstr\nU\x08\x00\x00\x00\x00\x00\x03\x1b\x9ctRU\x08\x03\xc6p\x12\xfc$Ogtt.'
It creates an Unpickler and then calls load().
103 def pickle_server_decode(msg):
104 """Decodes msg and returns its parts"""
105 unpickler = Unpickler(BytesIO(msg))
106 unpickler.find_global = server_find_global
107 try:
108 # PyPy, zodbpickle, the non-c-accelerated version
109 unpickler.find_class = server_find_global
110 except AttributeError:
111 pass
112
113 try:
--> 114 return unpickler.load() # msgid, flags, name, args
115 except:
116 logger.error("can't decode message: %s" % short_repr(msg))
117 raise
118
Note the find_global (hook?) that gets set up on the unpickler. When load is called we end up in server_find_global.
/Users/cutz/Documents/NextThought/nti.dataserver-buildout/eggs/ZEO-5.1.1-py2.7.egg/ZEO/asyncio/marshal.py(114)pickle_server_decode()
112
113 try:
--> 114 return unpickler.load() # msgid, flags, name, args
115 except:
116 logger.error("can't decode message: %s" % short_repr(msg))
/Users/cutz/Documents/NextThought/nti.dataserver-buildout/eggs/ZEO-5.1.1-py2.7.egg/ZEO/asyncio/marshal.py(165)server_find_global()
157 def server_find_global(module, name):
158 """Helper for message unpickler"""
159 try:
160 if module not in ('ZopeUndo.Prefix', 'copy_reg', '__builtin__'):
161 raise ImportError
162 m = __import__(module, _globals, _globals, _silly)
163 except ImportError as msg:
164 from IPython.core.debugger import Tracer; Tracer()()
--> 165 raise ImportError("import error %s: %s" % (module, msg))
166
167 try:
168 r = getattr(m, name)
169 except AttributeError:
170 raise ImportError("module %s has no global %s" % (module, name))
171
172 return r
ipdb>
Here module is zodbpickle and name is binary. Note the very first condition on line 160 raises if module is not in ('ZopeUndo.Prefix', 'copy_reg', '__builtin__'). This triggers the ImportError on line 161 to raise all the way up out of pickle_server_decode and its caller message_recieved (where the connection is closed).
It's worth noting that in server_find_global I am able to import zodbpickle successfully. It doesn't seems to be an issue with that. Rather it seems to be an issue with the fact that it is trying to load it from server_find_global when that module isn't allowed to be loaded that way?
from zeo.
Related Issues (20)
- tests hard depend on deprecated mock HOT 10
- test_ssl_hostname_check and test_ssl_basic fail
- Remove possibly unused MTAcceptor feature HOT 5
- ClientStorage: race condition when used with multiple addresses HOT 4
- Potential race condition indicated by `checkConcurrentUpdates2Storages` HOT 3
- (Commit-)`LockManager` is not fair
- Should we stop testing ZEO against all ZODB storage types? HOT 2
- Multiple ZEO data corruptions due to concurrency bugs HOT 13
- zeopack exit value should not be 0 when socket is not found
- asyncio tests are responding with `None` to register call HOT 1
- Race test `check_race_external_invalidate_vs_disconnect` fails on macOS HOT 4
- Client does not retry failed connections HOT 18
- Request release for 5.x branch on PyPI HOT 3
- Python 3.12 test failures HOT 2
- Upcoming test prefix related incompatibilities with ZODB HOT 1
- Look for more Python 2 leftovers HOT 1
- Release version 6.0 HOT 2
- ZEO.Exceptions.ServerException: (‘builtins.FileNotFoundError’, (2, ‘No such file or directory’)) in Plone 6.0.7 HOT 3
- AttributeError: 'FullGenericTests' object has no attribute 'getConfig' HOT 4
- Remove the `find_global`/`server_find_global` shims HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from zeo.