Comments (4)
In the end I went with Decimal and simplejson. Performance was not an issue.
I like jams, but I wanted to have as little overhead as possible.
Anyway, thanks for the help!
from jams.
Thanks @faroit for raising this one. It's an interesting and nontrivial problem.
One question I might ask up front: if you want an easy way to reduce file size, you can always export to jamz
(gzipped-jams). You gain significantly in file size, preserve accuracy, but lose human readability. Not an ideal solution in all scenarios, but it might work for you.
Otherwise, a couple of things to consider:
- As long as we produce valid json, anything we do to the encoder (including custom/extended encoder classes) is fair game.
- I wouldn't want to use custom data types to enforce quantization (Eg
decimal
), but if we could quantize on output by an optional kwarg, that'd be fine. I'm thinking something likejams.save(..., precision=3)
that gets threaded through all the encoding. - I'm not sure about simplejson, since the performance hit seems to be significant. See [benchmarks].
- I generally suspect that the desire for this particular feature is somewhat limited -- I for one always err on the side of precision, since disk is cheap, and you can always simplify the data when displaying to a user without permanently discarding anything. If you really want this kind of compression, it might be best to write a separate utility that post-processes json files to reduce numerical precision.
from jams.
fwiw, I once had an issue where json.dump()
would actually crash when trying to serialize a high precision float, and the solution was to use round(x, 15)
to limit the float to 15 decimals. So that might be a different reason to support a precision kwarg, although I have to say since then I haven't re-encountered this issue.
from jams.
fwiw, I once had an issue where json.dump() would actually crash when trying to serialize a high precision float,
That sounds like a json implementation error, and definitely above our pay grade (ie, tag-as-won't-fix).
from jams.
Related Issues (20)
- jam.validate() is True, but jam.save(path) crashes. Bug? HOT 2
- Feature Request: jams.convert(a, 'namespace'): from 'note_midi' to 'pitch_contour' HOT 7
- notebook rendering does not work in jupyterlab HOT 3
- sed_eval bindings? HOT 6
- sortedcontainers updates
- display(meta=False) by default? HOT 2
- support jsonschema >= 3.0 HOT 3
- load jams mapping error? HOT 5
- More than one curator? HOT 13
- on possible (minor) tweaks to Annotation.to_samples HOT 6
- jams.load does not seem to work with one of your example files HOT 3
- support pytest 4.0
- `jams.Annotation.trim()` doesn't include events with 0 duration that's on the trim boundary HOT 4
- __get_dtype fails when type is list
- Global observations: avoid time/duration HOT 8
- tag_medleydb_instruments points to a 404 error HOT 3
- Next generation jams HOT 6
- display.events labels and lines are misaligned HOT 1
- Install fails on Python 3.12: `imp` Module Missing HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from jams.