Comments (4)
I have recently discovered the xray package, which provides N-dimensional arrays (like numpy) but with named dimensions (like netCDF4 or pandas. It looks stable, well supported, and well documented. So I have decided to implement xray rather than create my own datatype(s) for this purpose.
from aospy.
I have begun to experiment with implementing xray support on a new branch [python3+xray]. At the moment it is fairly disjoint from the current develop branch (in that I have also migrated the code to python 3); there are also a lot of additions, many of which likely require some review and refactoring. That said it is functioning for my purposes at the moment.
I preserve all the code that works using the python-netCDF4 library to read in files, and numpy arrays for all computations. To account for this, I require the user to specify within a Run object how they would like things to be handled for that run with a new keyword argument (they can choose from either the read_mode='netcdf4'
or read_mode='xray'
tracks). There are basically just a number of if-statements within the code that depend on this argument, which keep the tracks separate; I feel like there must be a cleaner way, but for now it is quick and easy. With these modifications I can write calculation functions which take in DataArrays as arguments and return DataArrays as results. These results can then be saved directly as netCDF files rather than pickle files.
I apologize for taking a big leap here, but I wanted to be able to write calculation functions using DataArrays rather than numpy arrays from the start as I begin to develop my own library of functions (and not have to port them at some later time). I'm guessing that for the master version of aospy, we'll want to think about these issues a bit more carefully, but hopefully this is a decent start.
from aospy.
No apology necessary -- by all means, keep it coming. Getting both Python 3 support and xray implemented is huge. Let's meet in person to discuss the changes; want to make sure I understand it all. (Will email separately about that.)
I am inclined towards ultimately using xray exclusively, at least internally. Perhaps there is a clean way -- i.e. using decorators, isinstance
, and/or try/except clauses -- to automatically detect which datatype a particular user-specified function is expecting: if it is numpy arrays, then generate an xray object containing that array. (Not actually sure that would work -- haven't looked at the code in a while)
But you're right that, in the near term, we need to still support the existing functions. How most elegantly to support both, we should give some thought.
from aospy.
As of the most recent commit (3ebb63f), aospy uses xray almost exclusively within itself, accomplished by incorporating the code contributed by @spencerkclark.
The issue of converting functions that are used to compute physical quantities from handling numpy to xray objects is a separate one.
from aospy.
Related Issues (20)
- Error when passing non-default date ranges to `calc_suite_specs` HOT 6
- BOUNDS_STR and TIME_BOUNDS_STR HOT 14
- Daily output data HOT 1
- Perform calculations on subset of levels HOT 2
- recalculate HOT 1
- Calculations on only one variable HOT 3
- Towards v0.3 release HOT 15
- Use `stable` and `master` as our two main branches, rather than `master` and `develop` HOT 1
- Add support for zarr HOT 5
- Improve logging when calcs submitted in parallel
- Potentially use intake for describing/finding data on disk (i.e. what DataLoaders do)
- Failing tests in CI, but for some builds still come back as green
- New failure in test_apply_time_offset HOT 5
- Need to fix warnings due to recent updates to dependencies
- Move CI over to Azure pipelines HOT 1
- Use "black" tool for enforcing style
- Infer surface area from lat + lon if none provided. HOT 1
- Should we move aospy repository to its own 'aospy' organization? HOT 2
- Python errors in docs builds HOT 1
- YAML-based specification of aospy objects HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aospy.