Giter VIP home page Giter VIP logo

freedompp's Introduction

freedompp: post-processing tool xml-free

Install

python setup.py install

Quick start guide

This tool can be used either on the command line or interpreted in python. On the command line, run freedompp -h to see which options are available. Most common are choice of type (timeserie = ts, annual average = ann or monthly average = mm), field of interest, years to process and paths to history and pp directories.

  • Create a timeserie of so with:
freedompp -t ts -f so -c ocean_month_z -s 96 -e 100 -d /archive/myrun/history -o /archive/myrun/pp
  • Compute the monthly averages with:
freedompp -t mm -c ocean_month -s 96 -e 100 -d /archive/myrun/history -o /archive/myrun/pp

Other useful options include renaming the output component e.g. -r new_component_name, changing chunk sizes e.g. -K time 1 z_l 35, support for tiled output -N tile1.nc, split files, as well as various other overrides. For example:

freedompp -t ts -f thetao -c ocean_month_z -s 2018 -e 2018 \
          -d /archive/myrun/history -o /archive/myrun/pp \
          -K time 1 --prefix="" -R -N 4

recombines (-R) split files .nc.000[0-3] (-N/--nsplit=4) stored in the tar file without a prefix and write the recombined file with a chunking (-K/--chunk) of 1 time record.

By default, freedompp will load netcdf files from the tar files directly into memory. There is an option to write the history files to disk, using:

freedompp -t ts -f so -c D2ocean_month_z -s 2012 -e 2017 \
          -d /archive/myrun/history -o /archive/myrun/pp \
          -W -X /work/tmpdir

The package can also be used in interactive python environments, with function to load and write timeseries and averages.

  • load a timeserie in memory:
from freedompp.libfreedompp import load_timeserie
ts = load_timeserie('so', 'ocean_month_z', 96, 100,
                    historydir='/archive/myrun/history')
  • write a timeserie to disk:
from freedompp.libfreedompp import write_timeserie
write_timeserie('so', 'ocean_month_z', 96, 100,
                historydir='/archive/myrun/history',
                ppdir='/archive/myrun/pp')
  • compute monthly and annual averages:
from freedompp.libfreedompp import compute_average
monthly = compute_average('ocean_daily', 96,1 00, avtype='mm',
                          historydir='/archive/myrun/history')
annual = compute_average('ocean_daily', 96, 100, avtype='ann',
                         historydir='/archive/myrun/history')
  • or write them to disk:
from freedompp.libfreedompp import write_average
monthly = write_average('ocean_daily', 96, 100, avtype='mm',
                          historydir='/archive/myrun/history',
                          ppdir='/archive/myrun/pp')
annual = write_average('ocean_daily', 96, 100, avtype='ann',
                         historydir='/archive/myrun/history',
                         ppdir='/archive/myrun/pp')

and all aforementioned options are obviously available in python as well.

freedompp's People

Contributors

raphaeldussin avatar

Watchers

James Cloos avatar  avatar  avatar

Forkers

andrew-c-ross

freedompp's Issues

freedompp fails with `ValueError: buffer size must be a multiple of element size`

I want to extract ideal age variable (agessc) from downscaled annual z output. The component is named D2ocean_annual_z in the tar file. This is for the CM4X 1/8th historical run (CM4_historical_c192_OM4p125). I use freedompp on PP/AN as:

freedompp -t ts -f agessc -c D2ocean_annual_z -s 2010 -e 2010 -d /archive/Raphael.Dussin/FMS2019.01.03_devgfdl_20230608/CM4_historical_c192_OM4p125/gfdl.ncrc5-intel22-prod-openmp/history -o /archive/Jan-erik.Tesdal/FMS2019.01.03_devgfdl_20230608/CM4_historical_c192_OM4p125/gfdl.ncrc5-intel22-prod-openmp/pp

which results in the following error:

/nbhome/jet/miniconda3/bin/freedompp:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  __import__('pkg_resources').run_script('freedompp==0.0.1', 'freedompp')
Traceback (most recent call last):
  File "/nbhome/jet/miniconda3/bin/freedompp", line 4, in <module>
    __import__('pkg_resources').run_script('freedompp==0.0.1', 'freedompp')
  File "/nbhome/jet/miniconda3/lib/python3.9/site-packages/pkg_resources/__init__.py", line 722, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/nbhome/jet/miniconda3/lib/python3.9/site-packages/pkg_resources/__init__.py", line 1561, in run_script
    exec(code, namespace, namespace)
  File "/nbhome/jet/miniconda3/lib/python3.9/site-packages/freedompp-0.0.1-py3.9.egg/EGG-INFO/scripts/freedompp", line 185, in <module>
    write_timeserie(field, comesfrom, yearstart, yearend, **kwargs)
  File "/nbhome/jet/miniconda3/lib/python3.9/site-packages/freedompp-0.0.1-py3.9.egg/freedompp/libfreedompp.py", line 134, in write_timeserie
    ds, fids = open_files_from_archives(
  File "/nbhome/jet/miniconda3/lib/python3.9/site-packages/freedompp-0.0.1-py3.9.egg/freedompp/libIO.py", line 92, in open_files_from_archives
    ds = xr.open_mfdataset(open_files, **kwargs)
  File "/nbhome/jet/miniconda3/lib/python3.9/site-packages/xarray-2024.6.0-py3.9.egg/xarray/backends/api.py", line 1054, in open_mfdataset
    datasets = [open_(p, **open_kwargs) for p in paths]
  File "/nbhome/jet/miniconda3/lib/python3.9/site-packages/xarray-2024.6.0-py3.9.egg/xarray/backends/api.py", line 1054, in <listcomp>
    datasets = [open_(p, **open_kwargs) for p in paths]
  File "/nbhome/jet/miniconda3/lib/python3.9/site-packages/xarray-2024.6.0-py3.9.egg/xarray/backends/api.py", line 571, in open_dataset
    backend_ds = backend.open_dataset(
  File "/nbhome/jet/miniconda3/lib/python3.9/site-packages/xarray-2024.6.0-py3.9.egg/xarray/backends/scipy_.py", line 326, in open_dataset
    store = ScipyDataStore(
  File "/nbhome/jet/miniconda3/lib/python3.9/site-packages/xarray-2024.6.0-py3.9.egg/xarray/backends/scipy_.py", line 178, in __init__
    scipy_dataset = _open_scipy_netcdf(
  File "/nbhome/jet/miniconda3/lib/python3.9/site-packages/xarray-2024.6.0-py3.9.egg/xarray/backends/scipy_.py", line 126, in _open_scipy_netcdf
    return scipy.io.netcdf_file(filename, mode=mode, mmap=mmap, version=version)
  File "/nbhome/jet/miniconda3/lib/python3.9/site-packages/scipy/io/_netcdf.py", line 279, in __init__
    self._read()
  File "/nbhome/jet/miniconda3/lib/python3.9/site-packages/scipy/io/_netcdf.py", line 612, in _read
    self._read_var_array()
  File "/nbhome/jet/miniconda3/lib/python3.9/site-packages/scipy/io/_netcdf.py", line 724, in _read_var_array
    rec_array = frombuffer(self.fp.read(self._recs*self._recsize),
ValueError: buffer size must be a multiple of element size

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.