Comments (5)
One thing that can be done now is perhaps write out the version of Pyasdf that created the file.
Yes -- the size of the block header recently changed to support compression. Once we have a first stable numbered release of ASDF, then all versions of pyasdf going forward will support all versions of ASDF, but right now, I'm not dealing with such complications. The ASDF version number is at the top of every ASDF file, and each tag within an ASDF file is individually versioned, so this won't be difficult going forward. One of the major drawbacks of FITS is that it has never had a version number.
Why the big difference?
I suspect the difference is the inclusion of arrays in dist.asdf
, but I can't really say. Can you share the files with me?
from asdf.
On the performance issues. There was one obvious bug in pyasdf that was causing runtime to increase exponentially with the number of items in the tree. That should be fixed by #92.
After doing that, the breakdown on the image.asdf
example is essentially:
- 25% YAML parsing (in pyyaml)
- 25% JSON schema validation (about 25% of which is in pyasdf, the reset in jsonschema)
- 50% Construction of models
There may be some low-hanging fruit for optimization in models. And there's probably some extra in pyasdf, but not as much.
from asdf.
This sounds great. Thanks.
from asdf.
I'm making a little bit of progress on the construction of models. I just eliminated some overhead in creation of model classes, which happens every time you build a compound model. Will do better still once it's possible to convert an expression tree directly to a model class without making intermediate classes.
from asdf.
I'm going to close this because I think I've got all of the low-hanging performance fruit in pyasdf here.
from asdf.
Related Issues (20)
- Combine package and build workflows
- masked arrays do not roundtrip with all false masks
- `AsdfSpec` misses expected match
- deprecate `AsdfSpec` and `format_tag`
- Tracking `sunpy` 6.0 and ASDF 1.6.0 HOT 2
- Old (<2.14) versions of asdf do not fully support ASDF standard 1.6.0
- `AsdfFile` instances are not pickleable HOT 1
- Chunking support HOT 2
- Investigate enabling `validate_checksum` as default `True`
- Investigate returning `ndarray` when `lazy_load=False` HOT 8
- Change scope of ndarray custom validators HOT 1
- Add to docs comparison of `tag` vs `$ref` usage in schema.
- Consider a new design for the info and search methods that avoids conversion of nodes when the lazy_tree option is used
- Fix stable docs version in RTD
- FAILED asdf/_tests/test_yaml.py::test_implicit_conversion_warning HOT 3
- `assert_tree_match` and `np.testing` ignores array masks
- Schema_info Returning Non Schema Keyword HOT 2
- ignore_version_mismatch doesn't appear to be used? HOT 10
- Deprecate and remove the now unused `ignore_version_mismatch`
- A file with junk bytes after the tree leads to a read failure
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from asdf.