Following on @andersy005's discovery in #8 (comment), I just checked to see if the urls specified by this recipe are publicly available, and found that they are not.
In [1]: # copy and paste the `recipe.py` code in this repo here
In [2]: urls = [url for _, url in pattern.items()]
In [3]: urls[0]
Out[3]: 'https://tds.ucar.edu/thredds/fileServer/datazone/campaign/cesm/collections/ASD/hybrid_v5_rel04_BC5_ne120_t12_pop62/atm-regrid/proc/tseries/daily/FV_768x1152.bilinear.hybrid_v5_rel04_BC5_ne120_t12_pop62.cam.h1.LHFLX.00780101-00781231.nc'
In [4]: !wget 'https://tds.ucar.edu/thredds/fileServer/datazone/campaign/cesm/collections/ASD/hybrid_v5_rel04_BC5_ne120_t12_pop62/a
...: tm-regrid/proc/tseries/daily/FV_768x1152.bilinear.hybrid_v5_rel04_BC5_ne120_t12_pop62.cam.h1.LHFLX.00780101-00781231.nc'
...:
--2022-12-22 09:43:24-- https://tds.ucar.edu/thredds/fileServer/datazone/campaign/cesm/collections/ASD/hybrid_v5_rel04_BC5_ne120_t12_pop62/atm-regrid/proc/tseries/daily/FV_768x1152.bilinear.hybrid_v5_rel04_BC5_ne120_t12_pop62.cam.h1.LHFLX.00780101-00781231.nc
Resolving tds.ucar.edu (tds.ucar.edu)... 128.117.181.7
Connecting to tds.ucar.edu (tds.ucar.edu)|128.117.181.7|:443... connected.
HTTP request sent, awaiting response... 401
Username/Password Authentication Failed.
@paigem, as we discussed offline, I do recall that we wrestled with this issue initially. What I don't recall is if we solved it by finding an unauthenticaed endpoint for this data, or if the solution was that I just manually ran this recipe in a notebook the first time around (using your credentials).
Our discussion in #5 suggests to me that we did actually run this on Pangeo Forge Cloud (not in a notebook), because there would never have been a KilledWorker
in a notebook setting (at least as I would have done it). OTOH, I remember there was a long saga around getting pangeo-forge-recipes
to support passing credentials as query string parameters, which was motivated by this recipe. Which, on the contrary, makes me think there was never an unauthenticated pathway to this data. ๐คท
@paigem, bringing it back to what we can do now, are there currently any alternate endpoints for this data that do not require authentication? If the only option is to get this data via an authenticated endpoint, then we may be temporarily blocked here. Pangeo Forge Cloud does not currently have a mechanism for injecting data provider creds at runtime (other recipes blocked by this).
The tutorial notebooks provided in the docs demonstrate how to run pangeo-forge-recipes
yourself in a Jupyter notebook (as opposed to here, on Pangeo Forge Cloud):
https://pangeo-forge.readthedocs.io/en/latest/pangeo_forge_recipes/tutorials/index.html
this could be a possible alternative on the short term, given that you could provide your own credentials in that context.