Pickler's put_long method currently falls back on the text-based INT encoding if the long value is too large to be represented as a 4-byte signed integer.

Instead, I'm wondering whether it should use the LONG1 encoding and write it as an 8-byte signed integer. Since this method's parameter is a long I think all of the values should fit in a LONG1. My understanding is that LONG1 should be more time- and space-efficient for these values. Pyrolite already uses LONG1 encoding when writing BigIntegers.

If I use Pyrolite to do pickler.dumps(9223372036854775807L) (which is Long.MAX_VALUE), pickletools disassembles the result as:

    0: \x80 PROTO      2
    2: I    INT        9223372036854775807
   23: .    STOP
highest protocol among opcodes = 2

This matches Python 2.7's behavior.

In contrast, Python 3.7 pickles this value using LONG1 (which requires nearly half the space):

>>> pickletools.dis(pickle.dumps(9223372036854775807, protocol=2))
    0: \x80 PROTO      2
    2: \x8a LONG1      9223372036854775807
   12: .    STOP
highest protocol among opcodes = 2

Looks like decimal('NaN') is unsupported in net.razorvine.pickle:

Python pickle

# float('NaN') can be picled and loaded
>>> pickled = cloudpickle.dumps(float('NaN'))
>>> pickle.loads(pickled)
nan

# Decimal('NaN') can be pickled and loaded
>>> pickled = cloudpickle.dumps(decimal.Decimal('NaN'))
b'\x80\x05\x95!\x00\x00\x00\x00\x00\x00\x00\x8c\x07decimal\x94\x8c\x07Decimal\x94\x93\x94\x8c\x03NaN\x94\x85\x94R\x94.'
>>> pickle.loads(pickled)
Decimal('NaN')

Java net.razorvine.pickle

float('NaN') can be loaded as exepected

scala> unpickle.loads(PickleUtils.str2bytes("\u0080\u0005\u0095\n\u0000\u0000\u0000\u0000\u0000\u0000\u0000G\u007f\u00f8\u0000\u0000\u0000\u0000\u0000\u0000."))

Object = NaN

But Decimal('NaN') can NOT be loaded with java.lang.reflect.InvocationTargetException

scala> unpickle.loads(PickleUtils.str2bytes("\u0080\u0005\u0095!\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u008c\u0007decimal\u0094\u008c\u0007Decimal\u0094\u0093\u0094\u008c\u0003NaN\u0094\u0085\u0094R\u0094."))
net.razorvine.pickle.PickleException: problem construction object: java.lang.reflect.InvocationTargetException
  at net.razorvine.pickle.objects.AnyClassConstructor.construct(AnyClassConstructor.java:29)
  at net.razorvine.pickle.Unpickler.load_reduce(Unpickler.java:773)
  at net.razorvine.pickle.Unpickler.dispatch(Unpickler.java:213)
  at net.razorvine.pickle.Unpickler.load(Unpickler.java:123)
  at net.razorvine.pickle.Unpickler.loads(Unpickler.java:136)
  ... 48 elided

After some investigation, I found, as pointed from https://github.com/irmen/pickle/blob/master/README.md#python-to-java-unpickling , the decimal is mapped to java BigDecimal, but accroing to java doc the BigDecimal can't be NaN.

Do you have any idea for fixing it, Thanks!

Related: SPARK-36000 cc @HyukjinKwon @xinrong-databricks

I'm using pandas to save a dataframe with 'xz' compression.

df.to_pickle("abc.pickle", compression="xz")

and the dotnet version threw an exception while loading it.

Unhandled exception. Razorvine.Pickle.InvalidOpcodeException: invalid pickle opcode: 253

W're trying to load a imdb dataset which is packed in pickle format.
The loading sample is in the repo of TensorFlow.Keras.

The specific slowness code is:

(NDArray, NDArray) LoadX(byte[] bytes)
{
    var x = np.Load_Npz<int[,]>(bytes);
    return (x["x_train.npy"], x["x_test.npy"]);
}

You can use thie code snippet to reproduce this performance issue:

var cfg = new TransformerClassificationConfig();
var dataloader = new IMDbDataset(cfg); //the dataset is initially downloaded at TEMP dir, e.g., C:\Users\{user name}\AppData\Local\Temp\imdb\imdb.npz
var dataset = dataloader.GetData();

currently, integers are limited to the primitive long size.

don't forget to update the conversion table in the readme.

Hello,

I'm using sklearn2jpmml using this pickle library and facing the following error:

Exception in thread "main" java.lang.NegativeArraySizeException: -1394647443
    at net.razorvine.pickle.PickleUtils.readbytes(PickleUtils.java:54)
    at net.razorvine.pickle.Unpickler.load_binbytes(Unpickler.java:510)
    at net.razorvine.pickle.Unpickler.dispatch(Unpickler.java:315)
    at org.jpmml.python.PickleUtil$1.dispatch(PickleUtil.java:83)
    at net.razorvine.pickle.Unpickler.load(Unpickler.java:109)
    at org.jpmml.python.PickleUtil.unpickle(PickleUtil.java:104)
    at org.jpmml.sklearn.Main.run(Main.java:173)
    at org.jpmml.sklearn.Main.main(Main.java:161)

It happens when the model is "too big" apparently, is there known limitations about the size or is that a bug?

I'm using pandas, save a dataframe with 'xz' compression.

df.to_pickle("abc.pickle", compression="xz")

and dotnet version threw an exception.

Unhandled exception. Razorvine.Pickle.InvalidOpcodeException: invalid pickle opcode: 253

It is known that pickle fine created by python2 may not be loaded by python3 using "pickle.load"
Could your solution unpickle a binary file created by python2 involving Numpy arrays?

Got an exception saying that constructor needs 1 or 2 arguments but got 0 on load_reduce of a byte array.
Pickle file is as follows:
0: \x80 PROTO 2
2: } EMPTY_DICT
3: q BINPUT 0
5: ( MARK
6: X BINUNICODE 'modules/ATL.rpymc'
28: q BINPUT 1
30: ] EMPTY_LIST
31: q BINPUT 2
33: J BININT 1111638641
38: J BININT 1111622394
43: c GLOBAL 'builtin bytes'
62: q BINPUT 3
64: ) EMPTY_TUPLE
65: R REDUCE
66: q BINPUT 4
68: \x87 TUPLE3
69: q BINPUT 5
71: a APPEND

net.razorvine.pickle.PickleException: invalid escape sequence char 'U' in string "seth rollins \U0001f455 [...]" (possibly truncated)

Python pickles any unicode char above \uffff to \U0001xxxx.
decode_unicode_escaped() doesn't currently support
case: 'U'
and throws. Can this support be added?

I'm trying to load a wide serie of .sav files using your Nuget, but my attempts ended in this error.

expected zero arguments for construction of ClassDict (for numpy.dtype). This happens when an unsupported/unregistered class is being unpickled that requires construction arguments. Fix it by registering a custom IObjectConstructor for this class

Is it possible to set a default translation in a KeyValuePair<string, string> for the unregistered/unsupported classes, or add a specific constructor for numpy.dtype class?

DecimalConstructor.cs is calling

Convert.ToDecimal(stringArg, CultureInfo.InvariantCulture)

So something like

System.Convert.ToDecimal("5E-1",CultureInfo.InvariantCulture)

Will throw a System.FormatException: Input string was not in a correct format. exception

This has bit us when using Microsoft.Spark (which takes this library as a dependency). We build C# UDFs that run in Spark which requires data to be serialized back and forth between the JVM and the CLR. In sparkland we have some decimal values that Spark decides to represent in scientific notation and when this gets pickled into the CLR we hit the above issue.

I did a bit of research and using something like

Decimal.Parse("5E-1, NumberStyles.AllowAny)

would work but is maybe too permissive. I think at a minimum we would need

Decimal.Parse("5E-1, NumberStyles.AllowLeadingSign | NumberStyles.DecimalPoint | NumberStyles.Exponent)

but there might be others.

Please feedback if this is valid for loading pytorch torch. Save pickle format?