Issues with processing climate data about massbalance-sandbox HOT 5 OPEN

Hi Jordi,

It is possible that MBsandbox.process_era5_daily_data(gdir) and also MBsandbox.process_w5e5_data take some time to run because when running them for the first time, they first have to download the new (daily) data. So, depending on your internet connection, this can take some time. (e.g. in case of ERA5_daily it is 662MB, note that there only temperature is daily, and for precipitation the monthly estimates are used while for W5E5 e.g., both prcp and temp are daily ... in case of W5E5 it is 2*280MB). So, for testing daily MB, you should rather use W5E5, also because my preprocessed data goes until end of 2019 for W5E5 (and for ERA5 only goes till end of 2018).

You can also check manually inside of the folders whether the data has been correctly downloaded. In my case it is under: /home/lilianschuster/OGGM/download_cache/cluster.klima.uni-bremen.de/~lschuster/w5e5v2.0/flattened/daily (e.g. try to open it with xarray). If the download did not work, you could also manually download the files under this link here and drop them in the right folder. If it still does not work to open it with xarray and you can normally open stuff with xarray, then we know at least a bit better where the problem is.

I hope that helps a bit. This is all work in process, so if you encounter other problems, don't hesitate to ask further. Specifically the sfc type distinction model using a snow ageing bucket system is still not well documented.

from massbalance-sandbox.

Thanks for the quick reply! Now I'm getting a little bit more of information. The current situation is the following one:

When I run it using MBsandbox.process_w5e5_data(gdir, climate_type=climate, temporal_resol="daily") I get:

2021-09-30 17:46:00: oggm.cfg: PARAMS['hydro_month_nh'] changed from `10` to `1`.
2021-09-30 17:46:00: MBsandbox.mbmod_daily_oneflowline: (RGI60-11.01450) process_w5e5_data
2021-09-30 17:46:00: oggm.utils: No known hash for cluster.klima.uni-bremen.de/~lschuster/w5e5v2.0/flattened/daily/w5e5v2.0_tas_global_daily_flat_glaciers_1979_2019.nc
2021-09-30 17:46:00: oggm.utils: No known hash for cluster.klima.uni-bremen.de/~lschuster/w5e5v2.0/flattened/daily/w5e5v2.0_pr_global_daily_flat_glaciers_1979_2019.nc
2021-09-30 17:46:00: oggm.utils: No known hash for cluster.klima.uni-bremen.de/~lschuster/w5e5v2.0/flattened/daily/w5e5v2.0_glacier_invariant_flat.nc
2021-09-30 17:46:00: MBsandbox.mbmod_daily_oneflowline: OSError occurred during task process_w5e5_data on RGI60-11.01450: [Errno -101] NetCDF: HDF error: b'/Users/Bolib001/OGGM/download_cache/cluster.klima.uni-bremen.de/~lschuster/w5e5v2.0/flattened/daily/w5e5v2.0_tas_global_daily_flat_glaciers_1979_2019.nc'

Opening the file directly with xarray with xr.open_dataset("/Users/Bolib001/OGGM/download_cache/cluster.klima.uni-bremen.de/~lschuster/w5e5v2.0/flattened/daily/w5e5v2.0_tas_global_daily_flat_glaciers_1979_2019.nc") produces exactly the same error as above.

However, manually downloading the file from the server you sent me and opening with xarray works with no issues. To me, it looks like the issue comes from the automatically downloaded file from process_w5e5_data. FYI, I didn't have any issues when downloading the "classic" monthly CRU data from OGGM.

from massbalance-sandbox.

Hi Jordi,

Thanks for reporting back. I also just tested to delete the files and do process_w5e5_data: for me, it downloads the files normally and I can work with them afterwards. If this works for you, you can just go the manual way: via manually putting the files in the right folder, then when you do process_w5e5_data, oggm won't download the files again and you can then hopefully work with the climate files. But I will ask Fabien next week how to solve this. Probably this has something to do that I do not have any hash for that ... or, as you get an OSError, it might also be a problem from the path description, do you use linux, I have only tested it in linux systems?

from massbalance-sandbox.

What can happen is that the files are only partly downloaded if the user breaks the process while running, which could lead to the HDF errors reported here.

These files are probably not yet in the "verified" list (where we check for hashes) - probably we should add them soon

from massbalance-sandbox.

OK, so I manually placed the files in the default folder and it solved the issue. The automatically downloaded files might be somehow corrupted.

Thanks a lot for the help! Shall I keep the issue open?

from massbalance-sandbox.