Decimal('NaN') is unsupported in Java unpickling about pickle HOT 11 CLOSED

Thanks all!

from pickle.

Thank you, that's very helpful!

from pickle.

I do some local test, it works! It would be fixed in apache/spark#34285 in Spark.

Much thanks for your awesome help! @irmen

from pickle.

We can't make BigDecimal represent a NaN

I guess we can :

• map it into a float NaN
• or throw a better exception (NumberFormatException?)

I'll have a look if the first option is viable. However this means that you won't always get a BigDecimal object as a result of unpickling a python decimal. However I guess a Decimal NaN occurs very infrequently and if you need to support it it should be easy enough to test the type of the resulting object.

The same issue appears in the .NET version of the code btw.

from pickle.

@Yikun

Decimal("NaN") is now unpickled as Double.NaN. 2809d25

This is now released in pickle version 1.2.

from pickle.

@irmen Woow, thanks, might it posible backport this PR to https://github.com/irmen/Pyrolite and release a 4.32 version?

from pickle.

No, it would be a 5.2 version in that case because currently it's at 5.1. But since 5.0 pickle is no longer part of Pyrolite itself.
However I believe you’re only using the pickle part of pyrolite yeah? The way forward should be, I think, to replace pyrolite by just this pickle library. ?

from pickle.

@Yikun ping , not sure if you're read my reply above (because I haven't seen a change in the status of the ticked on the Spark issue tracker.)

from pickle.

Thanks for pinging @irmen! Seems like the backport isn't an option here, seems like the way we could try here is to replace to this pickle library.

from pickle.

@irmen Much thanks for your answer, I missed it before. And I will also try to switch pyrolite to pickle on spark upstream.

from pickle.

Yeah that should be the way to go. I don't believe you were using any of the non pickle related functions from the pyrolite library before so I think it should be fairly straightforward to switch?

from pickle.