Giter VIP home page Giter VIP logo

chains's Introduction

Hi there 👋

I am a climate scientist and engineer. Around GitHub, you'll find me contributing to open source projects like Xarray and Pangeo. I am the CTO at Earthmover, a startup building the cloud platform for scientific data teams. Previously, I was a co-founder and the technology director at CarbonPlan, a non-profit working on data science and policy issues surrounding carbon removal and climate solutions, and a project scientist in the Climate and Global Dynamics Laboratory at the National Center for Atmospheric Research. I'm a licensed Professional Engineer and I have a Ph.D. in Civil and Environmental Engineering from the University of Washington.

chains's People

Stargazers

avatar avatar

Watchers

avatar avatar avatar

Forkers

gutmann

chains's Issues

Test case broken: file concatenation preprocessing

I've tried to run the test case described in the storylines workflow and I get an error I don't understand :

$ snakemake hydrology_models --configfile ../storylines_test_data/test_config.yml
... a whole bunch of good output I can paste if you want...
Error in rule downscaling:
    jobid: 0
    output: /glade/scratch/gutmann/storylines/test_data/CNRM-CM5/hist/bcsd/downscaling_data/downscale.CNRM-CM5.hist.bcsd.nc
    log: /glade/scratch/gutmann/storylines/test_data/CNRM-CM5/hist/bcsd/logs/downscale.CNRM-CM5.hist.bcsd.20190312-145812.log.txt

RuleException:
ValueError in line 67 of /gpfs/fs1/work/gutmann/storylines/chains/downscaling.snakefile:
cannot broadcast shape (6, 2) to shape (365, 6, 365)
  File "/gpfs/fs1/work/gutmann/storylines/chains/downscaling.snakefile", line 176, in __rule_downscaling
  File "/gpfs/fs1/work/gutmann/storylines/chains/downscaling.snakefile", line 67, in process_downscaling_dataset
  File "/glade/u/home/gutmann/anaconda3/envs/storylines/lib/python3.6/site-packages/xarray/backends/api.py", line 658, in open_mfdataset
  File "/glade/u/home/gutmann/anaconda3/envs/storylines/lib/python3.6/site-packages/xarray/core/combine.py", line 552, in _auto_combine
  File "/glade/u/home/gutmann/anaconda3/envs/storylines/lib/python3.6/site-packages/xarray/core/combine.py", line 474, in _combine_nd
  File "/glade/u/home/gutmann/anaconda3/envs/storylines/lib/python3.6/site-packages/xarray/core/combine.py", line 492, in _auto_combine_all_along_first_dim
  File "/glade/u/home/gutmann/anaconda3/envs/storylines/lib/python3.6/site-packages/xarray/core/combine.py", line 510, in _auto_combine_1d
  File "/glade/u/home/gutmann/anaconda3/envs/storylines/lib/python3.6/site-packages/xarray/core/combine.py", line 510, in <listcomp>
  File "/glade/u/home/gutmann/anaconda3/envs/storylines/lib/python3.6/site-packages/xarray/core/combine.py", line 368, in _auto_concat
  File "/glade/u/home/gutmann/anaconda3/envs/storylines/lib/python3.6/site-packages/xarray/core/combine.py", line 122, in concat
  File "/glade/u/home/gutmann/anaconda3/envs/storylines/lib/python3.6/site-packages/xarray/core/combine.py", line 307, in _dataset_concat
  File "/glade/u/home/gutmann/anaconda3/envs/storylines/lib/python3.6/site-packages/xarray/core/variable.py", line 1978, in concat
  File "/glade/u/home/gutmann/anaconda3/envs/storylines/lib/python3.6/site-packages/xarray/core/combine.py", line 300, in ensure_common_dims
  File "/glade/u/home/gutmann/anaconda3/envs/storylines/lib/python3.6/site-packages/xarray/core/variable.py", line 1188, in set_dims
  File "/glade/u/home/gutmann/anaconda3/envs/storylines/lib/python3.6/site-packages/xarray/core/duck_array_ops.py", line 46, in f
  File "/glade/u/home/gutmann/anaconda3/envs/storylines/lib/python3.6/site-packages/dask/array/core.py", line 3228, in broadcast_to
  File "/glade/u/home/gutmann/anaconda3/envs/storylines/lib/python3.6/concurrent/futures/thread.py", line 56, in run
Exiting because a job execution failed. Look above for error message
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
An error occurred
Complete log: /gpfs/fs1/work/gutmann/storylines/chains/.snakemake/log/2019-03-12T145813.750351.snakemake.log

I copied over the test case storylines_test_data to my work dir, and changed the test_config.yml working dir from flash to a directory in my scratch space. The key error seems to be ValueError in line 67 of /gpfs/fs1/work/gutmann/storylines/chains/downscaling.snakefile: cannot broadcast shape (6, 2) to shape (365, 6, 365)
line 67 is :

    ds = xr.open_mfdataset(sorted(input_files),
                           preprocess=preproc,
                           engine='netcdf4').load()

Looking at the files, all seem to have a primary variable that is (time, lat, lon) and all seem to have the same lat, lon, bounds dimensions. All bounds_lat, bounds_lon variables seem to have the correct dimensions. The broadcast error above looks like there is a file with bounds_latitude(time, latitude, time), but I don't see anything like that. Any ideas?

Test case broken: renaming wind

After commenting out the preproc step in #1 it can successfully open files, but then it tries to rename the wind variable which doesn't exist in these files, does it need to create a constant wind=1-2m/s variable first :

...loading data...
renaming
<xarray.Dataset>
Dimensions:           (bound: 2, latitude: 6, longitude: 5, time: 54787)
Coordinates:
  * latitude          (latitude) float32 48.0625 48.1875 ... 48.5625 48.6875
  * longitude         (longitude) float32 -121.1875 -121.0625 ... -120.6875
  * time              (time) datetime64[ns] 1950-01-01 1950-01-02 ... 2099-12-31
Dimensions without coordinates: bound
Data variables:
    bounds_latitude   (time, latitude, bound) float64 48.0 48.12 ... 48.62 48.75
    bounds_longitude  (time, longitude, bound) float64 -121.2 -121.1 ... -120.6
    pr                (time, latitude, longitude) float32 18.118 12.0409 ... 0.0
    tasmax            (time, latitude, longitude) float32 -4.23 -8.21 ... -15.16
    tasmin            (time, latitude, longitude) float32 -12.67 ... -23.33
Attributes:
    Conventions:        GDT 1.2
    file_name:          conus_c5.cnrm-cm5_rcp45_r1i1p1.daily.pr.1950.nc
    History:            Archived Dec 2013
    authors:            Wood, A. and Mizukami, N., with codes from T. Pruitt,...
    description:        Daily 1/8th Degree CONUS Meteorology
    creation_date:      2013
    institution:        NCAR
    SurfSgnConvention:  Traditional
[Tue Mar 12 17:08:41 2019]
Error in rule downscaling:
    jobid: 0
    output: /glade/scratch/gutmann/storylines/test_data/CNRM-CM5/hist/bcsd/downscaling_data/downscale.CNRM-CM5.hist.bcsd.nc
    log: /glade/scratch/gutmann/storylines/test_data/CNRM-CM5/hist/bcsd/logs/downscale.CNRM-CM5.hist.bcsd.20190312-170812.log.txt

RuleException:
ValueError in line 73 of /gpfs/fs1/work/gutmann/storylines/chains/downscaling.snakefile:
cannot rename 'wind' because it is not a variable or dimension in this dataset
  File "/gpfs/fs1/work/gutmann/storylines/chains/downscaling.snakefile", line 176, in __rule_downscaling
  File "/gpfs/fs1/work/gutmann/storylines/chains/downscaling.snakefile", line 73, in process_downscaling_dataset
...a few more internal errors that probably aren't relevant...

Metsim error

Error in rule run_metsim_rcp:

RuleException:
CalledProcessError in line 61 of /gpfs/fs1/work/gutmann/storylines/chains/metsim.snakefile:
Command 'set -euo pipefail;  /glade/u/home/jhamman/miniconda3/envs/storylines/bin/ms -s distributed -n 1 /glade/scratch/gutmann/storylines/test_data/CNRM-CM5/rcp85/icar/configs/config.CNRM-CM5.rcp85.icar.metsim.60.cfg > /glade/scratch/gutmann/storylines/test_data/CNRM-CM5/rcp85/icar/logs/disagg.CNRM-CM5.rcp85.icar.metsim.60.20190314-135414.log.txt 2>&1' returned non-zero exit status 1.
  File "/gpfs/fs1/work/gutmann/storylines/chains/metsim.snakefile", line 61, in __rule_run_metsim_rcp
  File "/glade/u/home/gutmann/anaconda3/envs/storylines/lib/python3.6/concurrent/futures/thread.py", line 56, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
An error occurred
Complete log: /gpfs/fs1/work/gutmann/storylines/chains/.snakemake/log/2019-03-14T135411.527609.snakemake.log

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.