Giter VIP home page Giter VIP logo

cmip6-feedstock's Introduction

pangeo-forge-recipes

PyPI version CI Codecov Documentation Status Code style: black NSF Award 2026932

pangeo-forge is an open-source tool designed to aid the extraction, transformation, and loading of datasets. The goal of pangeo-forge is to make it easy to extract datasets from traditional data repositories and deposit them into cloud object storage in analysis-ready, cloud-optimized format.

pangeo-forge is inspired by conda-forge, a community-led collection of recipes for building Conda packages. We hope that pangeo-forge can play the same role for datasets.

Documentation

More can be learned about pangeo-forge, its progress, and related subprojects in its official documentation.

Contributing

pangeo-forge is still early in development - there are several ways to contribute:

  1. Create a recipe for a dataset you are interested in
  2. Open an issue or pull request here or in any of the related subprojects (pangeo-smithy, staged-recipes)
  3. Check out the project roadmap

Get in touch

Discussions on Pangeo Forge are generally hosted biweekly on Mondays at 2pm ET. Calendar link here. We aim to announce cancellations on this discourse thread.

License

This project is licensed under the Apache License, Version 2.0.

cmip6-feedstock's People

Contributors

cisaacstern avatar jbusecke avatar pangeo-forge-bot avatar

Stargazers

avatar avatar avatar

Watchers

avatar avatar avatar

Forkers

jbusecke cisaacstern

cmip6-feedstock's Issues

Request for additional variables

Hello,

I would like to access several variables using intake that aren't currently available in the pangeo archive.

Specifically, I need variables:
sidmassth
sidmassgrowthwat
friver

For source_id=HadGEM3-GC31-LL, experiment_id=ssp585, table_id=SImon

Is it possible to get those brought accross into the pangeo archive?

Max

CMIP6-PMIP4 request

Hi,
@cisaacstern suggested that I post an issue here (blame him).

For an NSF-funded project I am trying to obtain CMIP6-PMIP4 data for the past1000 or past2k runs (1000 or 1860y long at monthly resolution), extract surface temperature (tas), compute an appropriately weighted (cos(lat)) global mean, and export a data file (netCDF or other) with this for easy re-use and sharing.

As a use case, consider this simulation on the ESGF server. Would I be able to make it ARCO by tweaking recipe.py and replacing line 500 with CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn, or does the script need further massaging?

In case that makes a difference, the data are split between 93 files, each representing a 20-year period.

Once I figure out this particular recipe, the idea would be to give a similar fate to other PMIP4 transient simulations. I am new to pangeo-forge, but eager to learn.

Zooplankton data request

Hi,
I have been using the Pangeo Google Cloud to access CMIP6 data and used to request new data using the Google online form. I understand that Pangeo Forge is the new approach to assessing what is available and requesting new data on both GCS and AWS? I used the example for finding the instance_ids of the files I would love to see added to the cloud. These are zooplankton data, only 2D fields, and only a limited number of total CMIP6 models that output these. If it's possible to add these, I would be grateful. If I am misunderstanding how this works, please let me know.

Thank you!

Scenarios include historical, ssp126, ssp245, and ssp585:

['CMIP6.CMIP.MOHC.UKESM1-0-LL.historical.r19i1p1f2.Omon.zmeso.gn.v20200430',
 'CMIP6.CMIP.CCCma.CanESM5-CanOE.historical.r2i1p2f1.Omon.zmeso.gn.v20190429',
 'CMIP6.CMIP.MOHC.UKESM1-0-LL.historical.r4i1p1f2.Omon.zmeso.gn.v20200204',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r1i1p1f1.Omon.zmeso.gn.v20180803',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r14i1p1f1.Omon.zmeso.gn.v20180803',
 'CMIP6.CMIP.IPSL.IPSL-CM5A2-INCA.historical.r1i1p1f1.Omon.zmeso.gn.v20200729',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r27i1p1f1.Omon.zmeso.gn.v20191204',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r29i1p1f1.Omon.zmeso.gn.v20191204',
 'CMIP6.CMIP.CNRM-CERFACS.CNRM-ESM2-1.historical.r3i1p1f2.Omon.zmeso.gn.v20190125',
 'CMIP6.CMIP.CNRM-CERFACS.CNRM-ESM2-1.historical.r9i1p1f2.Omon.zmeso.gn.v20200117',
 'CMIP6.CMIP.MOHC.UKESM1-0-LL.historical.r10i1p1f2.Omon.zmeso.gn.v20200428',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r30i1p1f1.Omon.zmeso.gn.v20191204',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r33i1p1f1.Omon.zmeso.gn.v20211229',
 'CMIP6.CMIP.CNRM-CERFACS.CNRM-ESM2-1.historical.r11i1p1f2.Omon.zmeso.gn.v20200408',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r31i1p1f1.Omon.zmeso.gn.v20191204',
 'CMIP6.CMIP.CCCma.CanESM5-CanOE.historical.r3i1p2f1.Omon.zmeso.gn.v20190429',
 'CMIP6.CMIP.MOHC.UKESM1-0-LL.historical.r17i1p1f2.Omon.zmeso.gn.v20200430',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r25i1p1f1.Omon.zmeso.gn.v20180803',
 'CMIP6.CMIP.CCCma.CanESM5-CanOE.historical.r1i1p2f1.Omon.zmeso.gn.v20190429',
 'CMIP6.CMIP.CMCC.CMCC-ESM2.historical.r1i1p1f1.Omon.zmeso.gn.v20210114',
 'CMIP6.CMIP.MOHC.UKESM1-0-LL.historical.r9i1p1f2.Omon.zmeso.gn.v20200206',
 'CMIP6.CMIP.CNRM-CERFACS.CNRM-ESM2-1.historical.r6i1p1f2.Omon.zmeso.gn.v20200117',
 'CMIP6.CMIP.CNRM-CERFACS.CNRM-ESM2-1.historical.r7i1p1f2.Omon.zmeso.gn.v20200117',
 'CMIP6.CMIP.NOAA-GFDL.GFDL-ESM4.historical.r1i1p1f1.Omon.zmeso.gr.v20190726',
 'CMIP6.CMIP.MOHC.UKESM1-0-LL.historical.r6i1p1f3.Omon.zmeso.gn.v20200204',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r15i1p1f1.Omon.zmeso.gn.v20180803',
 'CMIP6.CMIP.MOHC.UKESM1-0-LL.historical.r11i1p1f2.Omon.zmeso.gn.v20200428',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r24i1p1f1.Omon.zmeso.gn.v20180803',
 'CMIP6.CMIP.MOHC.UKESM1-0-LL.historical.r8i1p1f2.Omon.zmeso.gn.v20200204',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r28i1p1f1.Omon.zmeso.gn.v20191204',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r10i1p1f1.Omon.zmeso.gn.v20180803',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r18i1p1f1.Omon.zmeso.gn.v20180803',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r17i1p1f1.Omon.zmeso.gn.v20180803',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r21i1p1f1.Omon.zmeso.gn.v20180803',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r19i1p1f1.Omon.zmeso.gn.v20180803',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r2i1p1f1.Omon.zmeso.gn.v20190305',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r12i1p1f1.Omon.zmeso.gn.v20180803',
 'CMIP6.CMIP.MOHC.UKESM1-0-LL.historical.r16i1p1f2.Omon.zmeso.gn.v20200430',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r26i1p1f1.Omon.zmeso.gn.v20180803',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r20i1p1f1.Omon.zmeso.gn.v20180803',
 'CMIP6.CMIP.MOHC.UKESM1-0-LL.historical.r7i1p1f3.Omon.zmeso.gn.v20200204',
 'CMIP6.CMIP.MOHC.UKESM1-0-LL.historical.r2i1p1f2.Omon.zmeso.gn.v20200309',
 'CMIP6.CMIP.CNRM-CERFACS.CNRM-ESM2-1.historical.r1i1p1f2.Omon.zmeso.gn.v20181206',
 'CMIP6.CMIP.CNRM-CERFACS.CNRM-ESM2-1.historical.r2i1p1f2.Omon.zmeso.gn.v20190125',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r4i1p1f1.Omon.zmeso.gn.v20180803',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r16i1p1f1.Omon.zmeso.gn.v20180803',
 'CMIP6.CMIP.MOHC.UKESM1-0-LL.historical.r1i1p1f2.Omon.zmeso.gn.v20200123',
 'CMIP6.CMIP.MOHC.UKESM1-0-LL.historical.r3i1p1f2.Omon.zmeso.gn.v20200319',
 'CMIP6.CMIP.CNRM-CERFACS.CNRM-ESM2-1.historical.r5i1p1f2.Omon.zmeso.gn.v20190125',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r22i1p1f1.Omon.zmeso.gn.v20180803',
 'CMIP6.CMIP.MOHC.UKESM1-0-LL.historical.r18i1p1f2.Omon.zmeso.gn.v20200430',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r32i1p1f1.Omon.zmeso.gn.v20190802',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r8i1p1f1.Omon.zmeso.gn.v20180803',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r23i1p1f1.Omon.zmeso.gn.v20180803',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r3i1p1f1.Omon.zmeso.gn.v20180803',
 'CMIP6.CMIP.MOHC.UKESM1-0-LL.historical.r5i1p1f3.Omon.zmeso.gn.v20200124',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r9i1p1f1.Omon.zmeso.gn.v20180803',
 'CMIP6.CMIP.CNRM-CERFACS.CNRM-ESM2-1.historical.r10i1p1f2.Omon.zmeso.gn.v20200117',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR-INCA.historical.r1i1p1f1.Omon.zmeso.gn.v20210216',
 'CMIP6.CMIP.CNRM-CERFACS.CNRM-ESM2-1.historical.r8i1p1f2.Omon.zmeso.gn.v20200117',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r5i1p1f1.Omon.zmeso.gn.v20180803',
 'CMIP6.CMIP.NIMS-KMA.UKESM1-0-LL.historical.r13i1p1f2.Omon.zmeso.gn.v20200805',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r13i1p1f1.Omon.zmeso.gn.v20180803',
 'CMIP6.CMIP.CNRM-CERFACS.CNRM-ESM2-1.historical.r4i1p1f2.Omon.zmeso.gn.v20190125',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r6i1p1f1.Omon.zmeso.gn.v20180803',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r7i1p1f1.Omon.zmeso.gn.v20180803',
 'CMIP6.CMIP.IPSL.IPSL-CM6A-LR.historical.r11i1p1f1.Omon.zmeso.gn.v20180803',
 'CMIP6.CMIP.MOHC.UKESM1-0-LL.historical.r12i1p1f2.Omon.zmeso.gn.v20200428',
 'CMIP6.ScenarioMIP.IPSL.IPSL-CM6A-LR.ssp126.r6i1p1f1.Omon.zmeso.gn.v20191121',
 'CMIP6.ScenarioMIP.IPSL.IPSL-CM6A-LR.ssp126.r14i1p1f1.Omon.zmeso.gn.v20191121',
 'CMIP6.ScenarioMIP.MOHC.UKESM1-0-LL.ssp126.r8i1p1f2.Omon.zmeso.gn.v20200415',
 'CMIP6.ScenarioMIP.CNRM-CERFACS.CNRM-ESM2-1.ssp126.r1i1p1f2.Omon.zmeso.gn.v20190328',
 'CMIP6.ScenarioMIP.IPSL.IPSL-CM6A-LR.ssp126.r4i1p1f1.Omon.zmeso.gn.v20191121',
 'CMIP6.ScenarioMIP.MOHC.UKESM1-0-LL.ssp126.r16i1p1f2.Omon.zmeso.gn.v20200710',
 'CMIP6.ScenarioMIP.MOHC.UKESM1-0-LL.ssp126.r5i1p1f2.Omon.zmeso.gn.v20201015',
 'CMIP6.ScenarioMIP.CMCC.CMCC-ESM2.ssp126.r1i1p1f1.Omon.zmeso.gn.v20210126',
 'CMIP6.ScenarioMIP.MOHC.UKESM1-0-LL.ssp126.r1i1p1f2.Omon.zmeso.gn.v20200213',
 'CMIP6.ScenarioMIP.MOHC.UKESM1-0-LL.ssp126.r19i1p1f2.Omon.zmeso.gn.v20200716',
 'CMIP6.ScenarioMIP.MOHC.UKESM1-0-LL.ssp126.r4i1p1f2.Omon.zmeso.gn.v20211202',
 'CMIP6.ScenarioMIP.CNRM-CERFACS.CNRM-ESM2-1.ssp126.r4i1p1f2.Omon.zmeso.gn.v20190410',
 'CMIP6.ScenarioMIP.NOAA-GFDL.GFDL-ESM4.ssp126.r1i1p1f1.Omon.zmeso.gr.v20180701',
 'CMIP6.ScenarioMIP.IPSL.IPSL-CM6A-LR.ssp126.r1i1p1f1.Omon.zmeso.gn.v20190903',
 'CMIP6.ScenarioMIP.MOHC.UKESM1-0-LL.ssp126.r12i1p1f2.Omon.zmeso.gn.v20200710',
 'CMIP6.ScenarioMIP.MOHC.UKESM1-0-LL.ssp126.r6i1p1f2.Omon.zmeso.gn.v20201009',
 'CMIP6.ScenarioMIP.CNRM-CERFACS.CNRM-ESM2-1.ssp126.r5i1p1f2.Omon.zmeso.gn.v20190410',
 'CMIP6.ScenarioMIP.CCCma.CanESM5-CanOE.ssp126.r1i1p2f1.Omon.zmeso.gn.v20190429',
 'CMIP6.ScenarioMIP.MOHC.UKESM1-0-LL.ssp126.r17i1p1f2.Omon.zmeso.gn.v20200710',
 'CMIP6.ScenarioMIP.MOHC.UKESM1-0-LL.ssp126.r9i1p1f2.Omon.zmeso.gn.v20200708',
 'CMIP6.ScenarioMIP.MOHC.UKESM1-0-LL.ssp126.r3i1p1f2.Omon.zmeso.gn.v20200310',
 'CMIP6.ScenarioMIP.CNRM-CERFACS.CNRM-ESM2-1.ssp126.r2i1p1f2.Omon.zmeso.gn.v20190410',
 'CMIP6.ScenarioMIP.MOHC.UKESM1-0-LL.ssp126.r10i1p1f2.Omon.zmeso.gn.v20200706',
 'CMIP6.ScenarioMIP.MOHC.UKESM1-0-LL.ssp126.r11i1p1f2.Omon.zmeso.gn.v20200710',
 'CMIP6.ScenarioMIP.CCCma.CanESM5-CanOE.ssp126.r3i1p2f1.Omon.zmeso.gn.v20190429',
 'CMIP6.ScenarioMIP.MOHC.UKESM1-0-LL.ssp126.r18i1p1f2.Omon.zmeso.gn.v20200716',
 'CMIP6.ScenarioMIP.CCCma.CanESM5-CanOE.ssp126.r2i1p2f1.Omon.zmeso.gn.v20190429',
 'CMIP6.ScenarioMIP.CNRM-CERFACS.CNRM-ESM2-1.ssp126.r3i1p1f2.Omon.zmeso.gn.v20190410',
 'CMIP6.ScenarioMIP.MOHC.UKESM1-0-LL.ssp126.r7i1p1f2.Omon.zmeso.gn.v20201015',
 'CMIP6.ScenarioMIP.IPSL.IPSL-CM6A-LR.ssp126.r2i1p1f1.Omon.zmeso.gn.v20190410',
 'CMIP6.ScenarioMIP.MOHC.UKESM1-0-LL.ssp126.r2i1p1f2.Omon.zmeso.gn.v20200310',
 'CMIP6.ScenarioMIP.IPSL.IPSL-CM5A2-INCA.ssp126.r1i1p1f1.Omon.zmeso.gn.v20220105',
 'CMIP6.ScenarioMIP.IPSL.IPSL-CM6A-LR.ssp126.r3i1p1f1.Omon.zmeso.gn.v20190410',
 'CMIP6.ScenarioMIP.IPSL.IPSL-CM6A-LR.ssp245.r11i1p1f1.Omon.zmeso.gn.v20191003',
 'CMIP6.ScenarioMIP.IPSL.IPSL-CM6A-LR.ssp245.r22i1p1f1.Omon.zmeso.gn.v20200218',
 'CMIP6.ScenarioMIP.MOHC.UKESM1-0-LL.ssp245.r2i1p1f2.Omon.zmeso.gn.v20200415',
 'CMIP6.ScenarioMIP.CNRM-CERFACS.CNRM-ESM2-1.ssp245.r2i1p1f2.Omon.zmeso.gn.v20190410',
 'CMIP6.ScenarioMIP.IPSL.IPSL-CM6A-LR.ssp245.r10i1p1f1.Omon.zmeso.gn.v20191003',
 'CMIP6.ScenarioMIP.IPSL.IPSL-CM6A-LR.ssp245.r5i1p1f1.Omon.zmeso.gn.v20191003',
 'CMIP6.ScenarioMIP.CNRM-CERFACS.CNRM-ESM2-1.ssp245.r8i1p1f2.Omon.zmeso.gn.v20201028',
 'CMIP6.ScenarioMIP.CMCC.CMCC-ESM2.ssp245.r1i1p1f1.Omon.zmeso.gn.v20210129',
 'CMIP6.ScenarioMIP.CCCma.CanESM5-CanOE.ssp245.r2i1p2f1.Omon.zmeso.gn.v20190429',
 'CMIP6.ScenarioMIP.IPSL.IPSL-CM6A-LR.ssp245.r25i1p1f1.Omon.zmeso.gn.v20200218',
 'CMIP6.ScenarioMIP.CNRM-CERFACS.CNRM-ESM2-1.ssp245.r6i1p1f2.Omon.zmeso.gn.v20201028',
 'CMIP6.ScenarioMIP.CNRM-CERFACS.CNRM-ESM2-1.ssp245.r1i1p1f2.Omon.zmeso.gn.v20190328',
 'CMIP6.ScenarioMIP.CNRM-CERFACS.CNRM-ESM2-1.ssp245.r10i1p1f2.Omon.zmeso.gn.v20201028',
 'CMIP6.ScenarioMIP.CCCma.CanESM5-CanOE.ssp245.r3i1p2f1.Omon.zmeso.gn.v20190429',
 'CMIP6.ScenarioMIP.IPSL.IPSL-CM6A-LR.ssp245.r2i1p1f1.Omon.zmeso.gn.v20190516',
 'CMIP6.ScenarioMIP.CNRM-CERFACS.CNRM-ESM2-1.ssp245.r5i1p1f2.Omon.zmeso.gn.v20190410',
 'CMIP6.ScenarioMIP.MOHC.UKESM1-0-LL.ssp245.r4i1p1f2.Omon.zmeso.gn.v20200721',
 'CMIP6.ScenarioMIP.IPSL.IPSL-CM6A-LR.ssp245.r14i1p1f1.Omon.zmeso.gn.v20191121',
 'CMIP6.ScenarioMIP.CNRM-CERFACS.CNRM-ESM2-1.ssp245.r9i1p1f2.Omon.zmeso.gn.v20201028',
 'CMIP6.ScenarioMIP.IPSL.IPSL-CM6A-LR.ssp245.r1i1p1f1.Omon.zmeso.gn.v20190119',
 'CMIP6.ScenarioMIP.IPSL.IPSL-CM6A-LR.ssp245.r3i1p1f1.Omon.zmeso.gn.v20191121',
 'CMIP6.ScenarioMIP.IPSL.IPSL-CM6A-LR.ssp245.r4i1p1f1.Omon.zmeso.gn.v20191121',
 'CMIP6.ScenarioMIP.CNRM-CERFACS.CNRM-ESM2-1.ssp245.r3i1p1f2.Omon.zmeso.gn.v20190410',
 'CMIP6.ScenarioMIP.IPSL.IPSL-CM6A-LR.ssp245.r6i1p1f1.Omon.zmeso.gn.v20191121',
 'CMIP6.ScenarioMIP.CNRM-CERFACS.CNRM-ESM2-1.ssp245.r7i1p1f2.Omon.zmeso.gn.v20201028',
 'CMIP6.ScenarioMIP.NOAA-GFDL.GFDL-ESM4.ssp245.r1i1p1f1.Omon.zmeso.gr.v20180701',
 'CMIP6.ScenarioMIP.CCCma.CanESM5-CanOE.ssp245.r1i1p2f1.Omon.zmeso.gn.v20190429',
 'CMIP6.ScenarioMIP.MOHC.UKESM1-0-LL.ssp245.r1i1p1f2.Omon.zmeso.gn.v20200415',
 'CMIP6.ScenarioMIP.MOHC.UKESM1-0-LL.ssp245.r3i1p1f2.Omon.zmeso.gn.v20200721',
 'CMIP6.ScenarioMIP.CNRM-CERFACS.CNRM-ESM2-1.ssp245.r4i1p1f2.Omon.zmeso.gn.v20190410',
 'CMIP6.ScenarioMIP.MOHC.UKESM1-0-LL.ssp245.r8i1p1f2.Omon.zmeso.gn.v20200721',
 'CMIP6.ScenarioMIP.IPSL.IPSL-CM6A-LR.ssp585.r3i1p1f1.Omon.zmeso.gn.v20191121',
 'CMIP6.ScenarioMIP.MOHC.UKESM1-0-LL.ssp585.r4i1p1f2.Omon.zmeso.gn.v20211201',
 'CMIP6.ScenarioMIP.NOAA-GFDL.GFDL-ESM4.ssp585.r1i1p1f1.Omon.zmeso.gr.v20180701',
 'CMIP6.ScenarioMIP.IPSL.IPSL-CM6A-LR.ssp585.r4i1p1f1.Omon.zmeso.gn.v20191122',
 'CMIP6.ScenarioMIP.IPSL.IPSL-CM6A-LR.ssp585.r33i1p1f1.Omon.zmeso.gn.v20211229',
 'CMIP6.ScenarioMIP.CNRM-CERFACS.CNRM-ESM2-1.ssp585.r3i1p1f2.Omon.zmeso.gn.v20190410',
 'CMIP6.ScenarioMIP.MOHC.UKESM1-0-LL.ssp585.r1i1p1f2.Omon.zmeso.gn.v20200415',
 'CMIP6.ScenarioMIP.MOHC.UKESM1-0-LL.ssp585.r2i1p1f2.Omon.zmeso.gn.v20200420',
 'CMIP6.ScenarioMIP.CNRM-CERFACS.CNRM-ESM2-1.ssp585.r4i1p1f2.Omon.zmeso.gn.v20190410',
 'CMIP6.ScenarioMIP.CNRM-CERFACS.CNRM-ESM2-1.ssp585.r1i1p1f2.Omon.zmeso.gn.v20191021',
 'CMIP6.ScenarioMIP.CNRM-CERFACS.CNRM-ESM2-1.ssp585.r2i1p1f2.Omon.zmeso.gn.v20190410',
 'CMIP6.ScenarioMIP.IPSL.IPSL-CM6A-LR.ssp585.r14i1p1f1.Omon.zmeso.gn.v20191121',
 'CMIP6.ScenarioMIP.CCCma.CanESM5-CanOE.ssp585.r3i1p2f1.Omon.zmeso.gn.v20190429',
 'CMIP6.ScenarioMIP.CMCC.CMCC-ESM2.ssp585.r1i1p1f1.Omon.zmeso.gn.v20210126',
 'CMIP6.ScenarioMIP.IPSL.IPSL-CM6A-LR.ssp585.r1i1p1f1.Omon.zmeso.gn.v20190903',
 'CMIP6.ScenarioMIP.CNRM-CERFACS.CNRM-ESM2-1.ssp585.r5i1p1f2.Omon.zmeso.gn.v20190410',
 'CMIP6.ScenarioMIP.MOHC.UKESM1-0-LL.ssp585.r3i1p1f2.Omon.zmeso.gn.v20200721',
 'CMIP6.ScenarioMIP.CCCma.CanESM5-CanOE.ssp585.r1i1p2f1.Omon.zmeso.gn.v20190429',
 'CMIP6.ScenarioMIP.CCCma.CanESM5-CanOE.ssp585.r2i1p2f1.Omon.zmeso.gn.v20190429',
 'CMIP6.ScenarioMIP.IPSL.IPSL-CM6A-LR.ssp585.r2i1p1f1.Omon.zmeso.gn.v20191121',
 'CMIP6.ScenarioMIP.IPSL.IPSL-CM6A-LR.ssp585.r6i1p1f1.Omon.zmeso.gn.v20191121',
 'CMIP6.ScenarioMIP.MOHC.UKESM1-0-LL.ssp585.r8i1p1f2.Omon.zmeso.gn.v20200721']

Dynamically detect netcdf version

In #8 we noticed that some of the datasets seem to be netcdf3 files.

This can be resolved easily by providing a keyword argument to file_pattern, but as with other kwargs, the sheer volume of recipes that this will ultimately generate makes that extremely tedious.

I would like to explore if there are ways to automatically determine the netcdf version from 1) the ESGF metadata 2) Or if there is a way to determine the netcdf format from a given url

I explored 1) with this:

url = "https://esgf-node.llnl.gov/esg-search/search"

params = {
            "type": "File",
            "retracted": "false",
            "replica":"false",
            "format": "application/solr+json",
            "latest": "true",
            "limit": 500,
            'variable_id':'thetao',
            'source_id':'GFDL-ESM4',
        }
resp = requests.get(url=url, params=params)
for k,v in resp.json()['response']['docs'][0].items():
    if ('cdf' in str(k)) or ('cdf' in str(v)):
        print(k)
        print(v)

But that only returns the actual url.

Is there a lightweight way to query the dataset and determine the version from the header?

Requesting data to be added to the cloud

I would love to have the following data added to the cloud, and I would prioritize it from most to least important as follows:

  1. 'CMIP6.*.*.*.hist-aer.*.Amon.pr.*.*'
  2. 'CMIP6.*.*.*.[hist-aer, hist-nat].*.Amon.ts.*.*'
  3. 'CMIP6.*.*.*.[hist-nat, hist-ghg].*.Amon.pr.*.*'
  4. 'CMIP6.*.*.*.hist-ghg.*.Amon.ts.*.*'
  5. 'CMIP6.*.*.*.piControl.*.Amon.ts.*.*',
  6. 'CMIP6.*.*.*.piControl.*.Amon.pr.*.*',
  7. 'CMIP6.*.*.*.piControl.*.fx.sftlf.*.*’
  8. 'CMIP6.*.*.*.historical.*.Amon.pr.*.*'
  9. 'CMIP6.*.*.*.historical.*.Amon.ts.*.*'

Thank you!

Slow recipe generation prevents scaling

Now that I am more seriously thinking about scaling this recipe up, I wanted to start a conversation about the performance issues I foresee.

Currently we generate a dictionary of recipes in serial via a for loop. Each recipe makes an API query and then generates some keyword arguments that need to be passed to the recipe generation.

If we want to scale this up to thousands or more of recipes we should spend some time understanding which parts of this machinery take the most time and how we could maybe avoid this.

Some preliminary thoughts:

  • Can we create the recipes in parallel or asyncronous?
  • Is there a way to 'batch' requests to the ESGF API.
  • Or could we work out some sort of 'cache' for recipes, so that only the ones that have changed get rebuilt/need to send an API request (I believe this is not trivially possible, since we need to build the recipe first to compare hashes @cisaacstern ?)

I intend this to be a loose conversation for now, so if anyone has ideas, please feel free to discuss.

Request for missing variables

Hi,

We have been doing some ocean analysis from CMIP6 models and found that there are some random missing fields in Pangeo cloud storage. Filling in those will help to complete the set of variables needed to do temperature and salinity budgets from around 20 models.
Using pangeo-forge-esgf tool, I could made a list of file ids which are available in ESGF, but not in cloud storage. Attaching the list. Can these data be uploaded into the cloud storage?

missing_file_ids.txt

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.