FrozenInstanceError with pickle in Spark about attrs HOT 13 CLOSED

So it turns out Spark's pickle does fancy things to pickle things pickle can't pickle. (say that fast).

Basic pickle couldn't serialize an attr.Attribute at all and mumbled something about getstate. Sure enough, you can fix the whole thing by adding these methods to Attribute:

      def __getstate__(self):
          """Play nice with pickle"""
          return tuple(getattr(self, a) for a in self.__slots__)

      def __setstate__(self, state):
          """Play nice with pickle"""
          __bound_setattr = _obj_setattr.__get__(self, Attribute)
          for name, value in zip(self.__slots__, state):
              __bound_setattr(name, value)

And it works with Spark.

Basic test:

x = attr.Attribute.from_counting_attr('x', attr.ib())
assert x == pickle.loads(pickle.dumps(x))

This fails with __getstate__ and succeeds with.
I'll send a pull request once I understand your test harness better. I'm guessing this goes in the dark magic section.

It would be a good idea to add this to frozen classes so they work with Pickle.

from attrs.

Well because 16.0 didn't have frozen classes? :) Pickle seems to use setattr therefore you can't use it with frozen classes.

from attrs.

Actually, my code doesn't use frozen. attr.Attribute uses frozen, and that's what breaks. This exact code is working fine with 16.0.0.

from attrs.

Oh drat!

from attrs.

(Commenting on the commit)

Ha, that's very clever. It cuts down attr.Attribute instantiation time by ~45%. However, it slows down access on the Attribute fields, so for example asdict is slower by ~12%. It's an interesting tradeoff, but I'd always trade class definition time (~once per process) for runtime operations like asdict that get called n times.

I wouldn't know about picking attr.Attribute, but pickling instances of non-frozen attr classes in general seems to work fine. (Frozen, not so much, yeah.) Does PySpark require you to pickle classes, as well as instances?

from attrs.

PySpark is more aggressive about what it pickles so it can support lambda and inline classes.

from attrs.

FWIW, I could live with adding PySpark to the test suite to prevent future regressions.

from attrs.

Just to clarify, my initial comment was to do with changing the Attribute class to have the _values tuple and properties. Adding __getstate__ and __setstate__ to frozen classes basically has no impact. From playing around in the REPL, it even works with nested attrs classes.

What are the exact combinations of parameters that trigger this? slots=True and frozen=True?

from attrs.

PR done. I added a test to pickle all the classes defined in the dark magic test. It seems to work fine with frozen classes without __getstate__. The problem was only slots classes.

It works fine with Spark, or rather, my previously working Spark code works with this patch.

from attrs.

Hm, really? On Python 3.5, only both slots=True and frozen=True fails. https://gist.github.com/Tinche/33b702c6a1c3e4093be95953720cd80c

from attrs.

I test with 2.7. Any slots=True failed. frozen=True, slots=False works fine.

from attrs.

Hm, can confirm, on 2.7 cases B and C fail.

On 3.5 only C fails.

from attrs.

Slots seems to be supported by Pickle in Python 3.5 without __getstate__. Anyway, my PR seems to cure all.

from attrs.