Support Decimal format/Quantization about jams HOT 4 CLOSED

In the end I went with Decimal and simplejson. Performance was not an issue.
I like jams, but I wanted to have as little overhead as possible.

Anyway, thanks for the help!

from jams.

Thanks @faroit for raising this one. It's an interesting and nontrivial problem.

One question I might ask up front: if you want an easy way to reduce file size, you can always export to jamz (gzipped-jams). You gain significantly in file size, preserve accuracy, but lose human readability. Not an ideal solution in all scenarios, but it might work for you.

Otherwise, a couple of things to consider:

• As long as we produce valid json, anything we do to the encoder (including custom/extended encoder classes) is fair game.
• I wouldn't want to use custom data types to enforce quantization (Eg decimal), but if we could quantize on output by an optional kwarg, that'd be fine. I'm thinking something like jams.save(..., precision=3) that gets threaded through all the encoding.
• I'm not sure about simplejson, since the performance hit seems to be significant. See [benchmarks].
• I generally suspect that the desire for this particular feature is somewhat limited -- I for one always err on the side of precision, since disk is cheap, and you can always simplify the data when displaying to a user without permanently discarding anything. If you really want this kind of compression, it might be best to write a separate utility that post-processes json files to reduce numerical precision.

from jams.

fwiw, I once had an issue where json.dump() would actually crash when trying to serialize a high precision float, and the solution was to use round(x, 15) to limit the float to 15 decimals. So that might be a different reason to support a precision kwarg, although I have to say since then I haven't re-encountered this issue.

from jams.

fwiw, I once had an issue where json.dump() would actually crash when trying to serialize a high precision float,

That sounds like a json implementation error, and definitely above our pay grade (ie, tag-as-won't-fix).

from jams.