Comments (6)
I've started to have a look at this but have come a potential issue:
Deriving lead_times
and reftime
both depend on having the start_date
(which is the same as the sub_experiment_id
). If this isn't in the netCDF file, I had been getting it from the ds_id
.
According to the initial rules we were given (roocs/34e-mngmt#98) it is possible that startdate
/sub_experiment_id
won't be in the file, so we can't depend on getting them from there.
This means that either a default time or the ds_id
need to be passed to the get_lead_times
/ get_reftime
functions, as we can't necessarily get it from the xarray dataset. At the moment the ds_id
isn't passed through to the fix functions (those in daops.data_utils
), so isn't available for me to use.
Options I have thought of are:
-
we could send the
ds_id
through to fix functions by default. I think this would help keep the fix templates generic. From a quick look this looks like its possible and relatively straightforward to do. -
we could have a way of passing arguments through somehow with the "derive: daops.data_utils.time_utils.function" string. This would the templates would be dataset specific so maybe isn't worth doing.
If we do the first option and pass the ds_id
through to the fix functions and implement this then there is the chance we might come across the same issue in future where the argument we need doesn't exist in the ds_id
and would need to passed through by the template.
@agstephens Hopefully that makes sense, what are your thoughts?
from daops.
Hi @ellesmith88, thinking about your first option...
we could send the
ds_id
through to fix functions by default. I think this would help keep the fix templates generic. From a quick look this looks like its possible and relatively straightforward to do.
I'm going to make some assertions (please tell me if I'm wrong):
- All fixes in
daops
anddachar
are only meaningful in the context of ads_id
. - Therefore, we will always have a
ds_id
available at the point where we want to look up the fix, in: https://github.com/roocs/daops/blob/master/daops/utils/core.py - There are no other places in the stack that would get fundamentally broken by changing the fix signature to:
fix(ds_id, ds, **operands)
If that is true, then I like your idea. It may well be useful in other cases of fixes in the future.
from daops.
I think all of your points are correct, I'll implement this and see how it goes
from daops.
Hi @ellesmith88, is this now implemented? Can we close it? Thanks
from daops.
@agstephens yes this is done.
from daops.
Thanks @ellesmith88
from daops.
Related Issues (20)
- Add features: CLI (subset only), Dockerfile and CWL file to the repo HOT 3
- order of elements of area needed for doc strings
- consolidate expects file path with DRS folder structure HOT 5
- Update test data management (daops) to use gitpython HOT 2
- Testing integration of daops as backend for cmip6_preprocessing HOT 1
- daops production error: ZeroDivisionError: float divmod() - with tiny (lat, lon) box HOT 5
- New intake catalog approach could speed up `consolidate` step
- IndexError: list index out of range HOT 3
- Process error: Cannot apply_along_axis when any iteration dimensions are 0 HOT 5
- Check all occurrences of `open_dataset` and `open_xr_dataset` use `roocs-utils` version of the function HOT 4
- Can we delete reference to "orchestrate"? HOT 1
- daops: subset-by-point
- daops.consolidate() is inefficient if lots of files - can we provide hints in the file path mapper? HOT 3
- daops consolidate - need to check why return types are different
- Fix lookup of catalogs in consolidate so that multiple catalogs could be used (e.g. cmip6 AND cmip5)
- Replicate content between ES indexes HOT 2
- Some CORDEX fixes have been captured by the py-cordex project
- Just send the command-line options directly into the `subset` function HOT 1
- Add unit tests for command-line HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from daops.