c2sm-rcm / cclm2cmor Goto Github PK

How useful are time averages of u and v (uas and vas)?
E.g. requested by EURO-CORDEX:
https://docs.google.com/spreadsheets/d/1qttqjJ9wmMTHWpSYdOMYArMQ-i12MhGY/edit#gid=1672965248

Rather use wind speed and wind direction?

Check if the script actually ensures that the output follows the CF conventions (http://cfconventions.org/Data/cf-conventions/cf-conventions-1.4/build/cf-conventions.pdf).
The following points are for instance copied from this document and not included in the CMOR script:

• Variable, dimension and attribute names should begin with a letter and be composed of letters, digits, and underscores.
• relative order of dimensions T,Z,Y,X. All other dimensions should, whenever possible, be placed to the left of the spatiotemporal dimensions.

At the moment there is quite some redundant code in the functions process_file, process_file_fix and proc_seasonal. A common function would be helpful for better maintenance.
Seasonal processing could be included in process_file.

Runoff values (aggregated 6h) are off. Check!

There are several points where performance in the shell scripts (first CMOR step) can be optimized:

• Include option to concatenate variables to yearly files directly from the one-timestep files (no intermediate monthly files).
  To allow using the CCLM Starter package (which produces monthly time series) it should, however, remain possible to work with monthly files in the second part of the shell scripts (second.sh)
• Write a python script that is executed from master_post.sh and that calls the shell scripts first.sh and second.sh. In this python script the processing of different years can be parallelized (with multiprocessing, similar as in the cmorlight.py python script)

Currently processing of each resolution starts with the high resolution input of the shell scripts (from now on called raw input), instead of using the output of the previous, lower resolution. This can be optimized with the following new structure:

• Sort resolutions by increasing time step size
• Start with highest resolution: full program in extra function on raw input
• Take output of this resolution for next lower resolution if the cell method (cm) is compatible with the previous one (otherwise process full program on raw input): most global and variable attributes should be still correct, process variable with cdo command and change time and time_bnds (create if previous resolution had cm 'point') variables, 'res' global attribute and filename and location.
  Note that also e.g. the cell method for seasonal/monthly processing 'maximum within days time: mean over days' is compatible with the daily cell method 'maximum'. In case of this special cell method, the script should check if the daily data are available with the correct cell method and if not create it at least temporarily (as is done now).
  If the seasonal processing uses the monthly output with cell method 'mean' the possibly unequal length of the different months should be taken into account.

Currently the snow area fraction, FR_SNOW (called snc in CORDEX), cannot be processed since it is neither a CCLM output variable nor is it calculated in the first CMOR processing step as is being done for some other variables.
The calculation of FR_SNOW should thus be included in this first processing step, e.g. in the file second.sh.
This is the formula needed for calculation of FR_SNOW taken from variables table:
Max(0.01,Min(1.,W_SNOW/0.015))*H(x)

I do not know, however, what H(x) is.

You can contact Klaus Keuler ([email protected]) on this issue.

If the variables table was not filled out correctly (missing or wrong fields) this is not always clear from the error messages. Include more error messages in this case or check the sanity of the variables table as a whole before processing.

• Include an option to perform chunking right after processing, such that the processing does not need to be called twice.
• Enable multiprocessing also for the chunking process: each core can create one chunked output file. The spawning should probably happen in tools.py