Comments (7)
As I understand it, ‘noload’ is dead in the python stdlib. However, that is a core requirement of ZODB and hence this library. (If you haven’t tried to GC a large database in an environment without ‘noload’, be prepared to wait. A lot )
from zodbpickle.
Well, I know all that. But there isn't any mention of the removal in the doc or the source code. In the Git history, cPickle.c was removed in 2007, and _pickle.c was added in 2008 (as part of https://bugs.python.org/issue2917), with useless commit messages.
Have we at least asked to get it back in Python 3 ?
from zodbpickle.
Even in Python 2.7, noload
is broken for ZODB. The code here uses its Python 2.6 behaviour (see #9), but that has Python bug 1101399. That bug doesn't matter for the ZODB use case, but it makes noload
less generic, which is a concern for the stdlib.
Given that Python 2.7 has a noload
, but it's broken for ZODB (and the commit that broke it refers to it as an "undocumented, obscure function" and didn't bother to add a test case), and Python 3.x has never had a noload, I would imagine the barrier to get it added to Python 3 would be fairly high. It would have to fix the same bug that 2.7 did but without breaking in the more general case (as well as be documented and tested, presumably).
The window for feature additions to Python 3.7 has closed, so at the earliest such an addition would be part of is 3.8 and we would need this library as long as ZODB supported any versions < 3.8.
I'm not saying it can't happen, but it sounds like a lot of work 😄 I would guess that a starting point would be posting to python-ideas.
from zodbpickle.
Thank you. I didn't know that the status was already so bad for Python 2.7.
In any case, we should also have a clear view of what we change, which is not easy when our code is based on old versions of the stdlib. A first step would be to sanitize our code, by having an upstream branch containing unmodified code of pickle from Python 3, and merge it from time to time to get new features and bugfixes. This way:
- we can easily see the remaining diff to try pushing upstream
- it would also help in providing protocol 4 for ZODB (#21)
from zodbpickle.
Recent discussion: #64 (comment)
from zodbpickle.
My current plan is to have three versions of the stdlib _python.c
fork in zodbpickle
:
- The current
_pickle_33.c
version will stick around only for folks needing extreme pickle-level Python2 compatibility: i.e., they have FileStorages which still contain (or might?) pickles generated under Python2, which must be loaded "correctly" under Python3. I'm not sure how users should signal their need for this version: an environment variable? Making an explicit function call? - A new
_pickle_38.c
fork, which adds back theUnpickler.noload
method, using the current Python 3.8load
as a template, and adds support for the new protocol 4 and 5 opcodes, as well as adopting the change to use module state. This version will be built for all Python3 versions < 3.12. - A new
_pickle_312.c
fork, applying similar transforms to restoreUnpickler.noload
, with additional changes to accommodate the fact that in 3.12, theUnpickler
type is heap-allocated. This version will be built for Python versions >= 3.12.
This change will also need to re-fork the pickletools
modules, to account for the new protocol support, and involves adjusting a bunch of tests.
I'm most of the way through the changes to the 3.8 version in C code, but wanted to get some discussion going on the approach before I go much further.
from zodbpickle.
See also #92 and the comment I made there about probably no longer needing _pickle_33.c
.
@tseaver Thank you for investing time into updating this library to recent changes in the pickle
universe.
from zodbpickle.
Related Issues (20)
- What's the reason behind `zodbpickle.binary`? HOT 2
- Should noload_ext1() et al instantiate an object? HOT 3
- Possible discrepency in Unpickler.noload_obj() HOT 1
- slowpickle.py assigns wrong unpickler
- Doctests in pickletools_*.py do not seem to be run
- ImportError: No module named pythainlp.tokenize.multi_cut HOT 4
- RFC: CPython 3.8.0b1 stdlib pickle.py depends on C _pickle module HOT 2
- PyPI wheels for Python 3.8 HOT 1
- Py2: binary class is large and tracked by gc; implement in C?
- Fails to build on Python 3.10.0a2 HOT 4
- SyntaxError !!! HOT 4
- Use PyFloat_Pack8() on Python 3.11 alpha7 HOT 1
- 2.3: python 2.x syntac in `zodbpickle` code HOT 2
- 2.3: pytest is failing HOT 9
- Can zodbpickle be forced to use dill? Currently it can't serialize custom classes like dill HOT 4
- RFE: is it possible to start making github releases?🤔 HOT 1
- Update `zodbpickle.Unpickler` to include the `find_global` support needed in ZEO HOT 1
- RFE: please prepare test suite to be able use it with pytest as well HOT 5
- When installing ZODB with pip, the zodbpickle failed building wheel HOT 1
- 'PURE_PYTHON=1' ignored during build
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from zodbpickle.