monocongo / climate_learn Goto Github PK

Develop an initial Pipeline for managing machine learning workflow.

Determine which model works best for features: [PS, T, U, and V] and label: PTTEND

PTTEND (T total physics tendency) is the result of the CAM's processing of the PS (surface pressure), T (temperature), U (zonal wind), and V (meridional wind) input variables. This is a linear function.

We assume this process is suited to a simple linear regression model.

Score various linear models and corresponding parameters to determine which works best for these features/label.

We've been doing scaling erroneously using a scaler for each lat/lon location, rather than a scaler per feature/label.

Create a simple CNN with Keras to evaluate its performance and predict variable values.

Use xarray.open_mfdataset() to facilitate multiple input files as a single dataset.

SHFLX (surface sensible heat flux) and LHFLX (surface latent heat flux) are results of the CAM's processing of the Q (specific humidity) and PRECL (large-scale (stable) precipitation rate (liquid + ice)) input variables. This is a non-linear function.

Score various non-linear models and corresponding parameters to determine which works best for these features/labels.

Create PBS batch processing script for running a ML model using CAM inputs/outputs.

Script template:

####  PBS preamble

#PBS -N PBS_test_script
#PBS -M [email protected]
#PBS -m abe

#PBS -A example_flux
#PBS -l qos=flux
#PBS -q flux

#PBS -l nodes=4:ppn=2,pmem=2gb
#PBS -l walltime=1:15:00
#PBS -j oe
#PBS -V

####  End PBS preamble

if [ -s "$PBS_NODEFILE" ] ; then
    echo "Running on"
    cat $PBS_NODEFILE
fi

if [ -d "$PBS_O_WORKDIR" ] ; then
    cd $PBS_O_WORKDIR
    echo "Running from $PBS_O_WORKDIR"
fi

#  Put your job commands after this line
echo "Hello, world."

The NCAR Community Atmospheric Model that we're basing some of this work upon operates in the same way on the northern and southern hemispheres, so we can use this as a convenient split for train/test datasets, as opposed to using scikit-learn's train_test_split() function that we've been using for train/test splits.

We now have access to new data sets for our project that use a more complicated
(non-linear) forcing mechanism for temperature and also for a new moisture variable ‘Q’.

The new state variables in the ‘h0’ (feature/input) files are ‘Q’ and the large-scale rainfall rate ‘PRECL’ as a diagnostic quantity. The ‘h1’ (output/target) files now contain the new forcing ‘PTEQ’ for the moisture variable ‘Q’. Basically, PTEQ is the time tendency of the moisture, just like the time tendencies for T, U, V.

We'll incorporate this new feature variable into existing fits/tests as well as adding new model types to explore how these will perform.

We currently expect data for prediction to match the shape of the fit inputs, i.e. there is currently no allowance for batching in the model(s).

Using Keras with the TensorFlow backend we'll create a simple artificial neural network (ANN) using two layers, 1) convolutional neural network and 2) LSTM, for predicting the CAM PTTEND forcing tendency variable from PS, T, U, and V inputs. We'll split our input data into train/test datasets on northern and southern hemispheres.

Add information to README regarding approach and methodologies used for building model to reproduce results of CAM for global climate modeling using ML algorithms.

Let's try using apricot for feature selection and/or dataset reduction (selection of "submodules") for model train/fit.

Create a script that

• reads in features and labels (preferably multiple inputs with xarray.open_mfdataset
• trains/fits one or more models using these inputs
• predicts targets using the fitted model(s)
• writes results as NetCDF, including differences compared against original CAM target results

See the guidance document for Travis CI.

• develop (unit) tests and mechanism for launch of all tests (minimally requires #7)
• add a .travis.yml file
• register project repository with Travis

Begin development of a notebook that includes an RNN/LSTM model implemented using Keras.