Moderately large BTrees fail to pickle with `RuntimeError: maximum recursion depth exceeded` about btrees HOT 10 OPEN

I'm running into this right now. There was some work, eight years ago, to have pickle not use python's call stack, but instead use a deque object. This would have allowed for pickling of deeply nested structures. http://bugs.python.org/issue3119

I'm looking to see if anyone has an implementation of this idea, and I'll update this issue if I find one.

from btrees.

Alternately, it might be possible do take an approach similar to https://github.com/google/pygtrie/blob/master/pygtrie.py#L165-L252 that flattens out the structure in __getstate__ and rehydrates in the __setstate__.

@jamadden, @tseaver would you all consider such an approach. It would be very good for folks like me that want to use BTrees outsize of zope. But, I'm not sure if it would be a meaningful deoptimzation for zope.

If you think it's a possible path, I might start working on a PR.

from btrees.

@fgregg I would be worried about any "intrusive" change to make pickling outside ZODB work: the machinery is tricky. Perhaps another tree implementation would be a better choice? E.g.: bintrees.

from btrees.

That makes complete sense, @tseaver. Because I am taking advantage of some of the facilities of zope.index, I might purse a fork of BTrees. If I get something working, we can revisit the question with concrete examples.

from btrees.

I wonder what use cases is motivating this.

from btrees.

Hi Jim, Here's my use case:
http://dedupe.readthedocs.io/en/latest/Making-smart-comparisons.html#index-blocks

On Fri, Nov 11, 2016 at 8:09 AM Jim Fulton [email protected] wrote:

I wonder what use cases is motivating this.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#44 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAgxbWC-_ZGnAATbZYHk8gV9phBc9ZhYks5q9HcTgaJpZM4JtRZ9
.

from btrees.

There's nothing in the document that indicates to me a need to pickle a BTree.

Is there some implicit assumption that you aren't using ZODB? If so, why dat? :)

from btrees.

No, I'm not using ZODB, because I've never gotten it to work with my
unusual use case.

On Fri, Nov 11, 2016 at 9:27 AM Jim Fulton [email protected] wrote:

There's nothing in the document that indicates to me a need to pickle a
BTree.

Is there some implicit assumption that you aren't using ZODB? If so, why
dat? :)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#44 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAgxbVg7-Dww6Kh5jm3dME-FbCIpXDKqks5q9IlHgaJpZM4JtRZ9
.

from btrees.

That's a bummer. If you feel like mentioning the issues you ran into, maybe on the ZODB list, I'd be interested to see if Ican help.

ZODB (or some of its machinery) seem like a good fit for this use case as they inherently break BTrees up and pickle the parts separately. It's a little hard to say though because I don't know what you're doing with the pickle.

from btrees.

If ZODB isn't involved, it it practical to pickle BTree items?

from btrees.