HRRR as Zarr on AWS about herbie HOT 21 CLOSED

@johnhorel, I finished my processing of the HRRR 2019 data. I created a "best time series" with my variables of interest as a single Zarr dataset. I chunked the (time=8760, y=1059, x=1799) arrays into (time=144, y=300, x=300) chunks (50MB).

This allows for reasonable access times whether acquiring the entire US at a single time step, or obtaining the entire archive time series at a specified location.

Here's a few screengrabs from this example analysis/visualization notebook:

To create this single cloud-optimized Zarr dataset from the initial GRIB2 files, I used basically three steps, captured in these notebooks:

1. Download
2. Fill Gaps
3. Rechunk

My documentation is a bit light but I'd be happy to discuss further here if there is interest.

Ping @ktyle , @abarciauskas-bgse, @zflamig, @ocefpaf

from herbie.

Yep, several months in zarr format are available already in AWS behind the curtain. We are grateful to @zflamig and the AWS Open Data Program to make that possible. We'll let you know as soon as the project goes live. Taylor Gowan is working on example codes for people to start from. Here's a taste to show the 96 chunks. Dang- you won't be in just one chunk, but stitching a few together is not a big deal.

from herbie.

from herbie.

Yes, there is a directory structure to tunnel down until you get the specific field/vertical level/YYMMDDHH but then in one directory there are 96 chunks within which are the all forecasts for that model run so it is easier to do data analytics. It will be straightforward to grab what you need with the understanding of how complex NCEP has the naming convention for types of levels, variables, etc.

from herbie.

Hi!
Not the wrong place to ask. I'm no longer at the University of Utah working on this project, but yes, that is the plan! And I'm excited to see it happen. As far as I understand, the work is currently underway and a big chunk has already been moved to AWS. @zflamig @johnhorel @mesowx would know more than I do about that effort.

from herbie.

@johnhorel , that's super cool! You mean those are the chunks within the single Zarr dataset, right?

from herbie.

@johnhorel, I'm trying to create a "best time series" from HRRR for the purpose of providing boundary conditions for coastal ocean models. I downloaded all the F01 hour data from AWS in grib format for 2019 as a test, but found there were 42 missing files:

fmissing = ['noaa-hrrr-bdp-pds/hrrr.20190310/conus/hrrr.t02z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190310/conus/hrrr.t03z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190310/conus/hrrr.t04z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190310/conus/hrrr.t05z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190310/conus/hrrr.t06z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190310/conus/hrrr.t07z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190310/conus/hrrr.t08z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190310/conus/hrrr.t09z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190310/conus/hrrr.t10z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190310/conus/hrrr.t11z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190310/conus/hrrr.t12z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190310/conus/hrrr.t13z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190310/conus/hrrr.t14z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190310/conus/hrrr.t15z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190310/conus/hrrr.t16z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190310/conus/hrrr.t17z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190310/conus/hrrr.t18z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190310/conus/hrrr.t19z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190310/conus/hrrr.t20z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190310/conus/hrrr.t21z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190310/conus/hrrr.t22z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190310/conus/hrrr.t23z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190311/conus/hrrr.t13z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190311/conus/hrrr.t14z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190311/conus/hrrr.t15z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190311/conus/hrrr.t16z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190503/conus/hrrr.t23z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190504/conus/hrrr.t10z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190504/conus/hrrr.t11z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190504/conus/hrrr.t12z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190504/conus/hrrr.t13z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190504/conus/hrrr.t14z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190504/conus/hrrr.t18z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190504/conus/hrrr.t19z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190504/conus/hrrr.t20z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190504/conus/hrrr.t21z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190504/conus/hrrr.t22z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190504/conus/hrrr.t23z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190906/conus/hrrr.t08z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20190906/conus/hrrr.t09z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20191122/conus/hrrr.t17z.wrfsfcf01.grib2',
 'noaa-hrrr-bdp-pds/hrrr.20191122/conus/hrrr.t18z.wrfsfcf01.grib2']

Can you confirm that those are indeed missing?
If so, what actions would you recommend to create a best time series without gaps?

from herbie.

from herbie.

from herbie.

from herbie.

@johnhorel , I checked Google Cloud, and the same 42 files seem to be missing. 😞

from herbie.

Wonder if these gaps correspond to instances where the NCEP model production suite had problems that forced runs to be scrubbed.

from herbie.

@ktyle, yipes, it didn't occur to me that these might be actual gaps in the forecast -- I assumed they were just files that failed to transfer at some point in the workflow.

@johnhorel , if it's not going to be possible to recover these files, could you please let me know so I can develop a workaround filling in with other forecast hours from the last good forecasts before the gaps?

from herbie.

@johnhorel, I wrote a script to fill the gaps with the best available data from previous long forecasts (from the forecasts at 0, 6, 12, 18 hours).

But if the missing original grib files are actually available, I'll use those instead!

from herbie.

from herbie.

Is there a way to track the availability in Zarr format? Or see some examples of the conversion?

I'm interested in applying Uber's H3 library to downsample the forecasts and allow accessibility for wider geographic areas without having to download and interpolate the full resolution forecast products.

from herbie.

from herbie.

@johnhorel Just sent you a personal email.

from herbie.

AWS announced HRRR as a new public dataset on January 14, 2021: https://aws.amazon.com/about-aws/whats-new/2021/01/new-aws-public-datasets-available/

GRIB2 bucket explorer: https://noaa-hrrr-bdp-pds.s3.amazonaws.com/index.html
Zarr bucket explorer: https://hrrrzarr.s3.amazonaws.com/index.html

from herbie.

@taylorgowan, I'm hoping this example of whole domain as Zarr is useful. Shout if you would like something different!

from herbie.

from herbie.