Comments (2)
bouweandela
commented on July 20, 2024
I'm no expert on this, so I cannot answer all your questions. Maybe @agstephens can tell you more?
I did notice the following though: the experiment facet is called experiment_id and ensemble is either called variant_label or member_id (I'm not sure about the difference) for CMIP6. When I run
conn = SearchConnection('https://esgf-node.llnl.gov/esg-search')
ctx = conn.new_context(project='CMIP6', experiment_id='past1000', variable='tas')
print('Hits: {}, Realms: {}, Ensembles: {}'.format(
ctx.hit_count,
ctx.facet_counts['realm'],
ctx.facet_counts['member_id']))
print(ctx.get_facet_options())
the following results are shown:
Hits: 18, Realms: {'atmos': 18}, Ensembles: {'r1i1p1f2': 4, 'r1i1p1f1': 14}
{'access': {'LAS': 5, 'GridFTP': 11, 'Globus': 9}, 'creation_date': {'2022-05-26T10:39:28Z': 2, '2020-04-21T10:17:53Z': 1, '2020-02-10T07:14:54Z': 1, '2020-02-10T06:30:46Z': 1}, 'data_node': {'esgf3.dkrz.de': 2, 'esgf.ceda.ac.uk': 1, 'esgf-data2.llnl.gov': 1, 'esgf-data2.diasjp.net': 3, 'esgf-data1.llnl.gov': 5, 'esgf-data04.diasjp.net': 1, 'esgf-data03.diasjp.net': 2, 'esgf-data02.diasjp.net': 1, 'esg-dn1.nsc.liu.se': 2}, 'data_specs_version': {'01.00.31': 11, '01.00.29': 3, '01.00.28': 4}, 'datetime_end': {'1850-12-16T00:00:00Z': 1, '1849-12-16T00:00:00Z': 2, '1849-04-18T00:00:00Z': 2}, 'frequency': {'mon': 13, 'day': 5}, 'grid': {'native atmosphere TL159 gaussian grid (160x320 latxlon)': 8, 'native atmosphere T42 Gaussian grid': 4, 'gs2x1.5': 3, 'T159L62': 3}, 'grid_label': {'gr1': 3, 'gr': 3, 'gn': 12}, 'index_node': {'esgf.ceda.ac.uk': 1, 'esgf-node.llnl.gov': 13, 'esgf-data.dkrz.de': 2, 'esg-dn1.nsc.liu.se': 2}, 'institution_id': {'MRI': 8, 'MIROC': 4, 'INM': 3, 'EC-Earth-Consortium': 3}, 'member_id': {'r1i1p1f2': 4, 'r1i1p1f1': 14}, 'model_cohort': {'Registered': 13, 'Published': 4}, 'nominal_resolution': {'500 km': 4, '100 km': 14}, 'short_description': {'MIROC-ES2L output prepared for CMIP6': 2, 'INM-CM4-8 output prepared for CMIP6': 2, 'EC-Earth3-Veg-LR output prepared for CMIP6': 1}, 'source_id': {'MRI-ESM2-0': 8, 'MIROC-ES2L': 4, 'INM-CM4-8': 3, 'EC-Earth3-Veg-LR': 3}, 'source_type': {'CHEM': 8, 'BGC': 4, 'AER': 15}, 'table_id': {'day': 5, 'Amon': 13}, 'variant_label': {'r1i1p1f2': 4, 'r1i1p1f1': 14}, 'version': {'20220526': 3, '20200423': 3, '20200318': 4, '20200120': 8}}
For bulk downloading data from ESGF I've been developing a more user-friendly alternative that is based on the esgf-pyclient package for the ESMValTool community. It is available as two Python functions, one called esmvalcore.esgf.find_files for searching for files and one called esmvalcore.esgf.download for downloading the results in parallel. They can be configured through a configuration file. Some of the advantages of these functions are over plain ESGF search are
- they automatically fall back to another server if one is offline
- they try to deal with some of the issues caused by wrong facet values on ESGF
- they can download multiple files in parallel
- they offer a unified facet naming scheme for the different projects, the mapping to the names used on ESGF is available here
If you're interested in trying those and run into trouble, feel free to ask me for help.
from esgf-pyclient.
d-otto
commented on July 20, 2024
Ahh thank you! This is an extremely helpful reply! Thank you for pointing out my issue :~) Thanks for the excellent software as well, I'm glad someone is working on smoothing out this whole process in a more maintainable way. I haven't worked with esmvaltool before but I'll definitely be looking into it!
from esgf-pyclient.
Related Issues (20)
- Failing CI build: parsing of HTTPResponse by defusedxml.ElementTree HOT 1
- No consistent result from different queries (CMIP6) HOT 1
- `ignore_facet_check` search option appears to be broken
- Dependency problem (version mismatch between pyesgf and requests library) leading to AttributeError: module 'requests_cache' has no attribute 'core' HOT 5
- add "facets" keyword argument to DatasetResult.file_context
- New release? HOT 5
- Import trial of `MyProxyClient` in `pyesgf/logon.py` outputs misleading error and incompatibility with `cryptography` from Anaconda `main` channel HOT 2
- logon for http request HOT 2
- lm.logon timeout HOT 12
- Facets warning with aggregation_context(): unexpected keyword
- logon does not allow access to all ESGF nodes HOT 13
- Unexpected number of results for large query
- Not matching all the expected files HOT 4
- logon refused from my laptop HOT 3
- Facets warning appears even when facets are specified HOT 5
- collection import errors from python 3.10 HOT 1
- [via MyProxyClient] Import of `MyProxyClient` fails due to complete retirement of `SafeConfigParser` in Python 3.12 HOT 2
- Fix `tests/test_results` test module
- 503 HTTPErrors should not be failing the tests
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from esgf-pyclient.