ecmwf / pyodc Goto Github PK

View Code? Open in Web Editor NEW

6.0 17.0 4.0 269 KB

Python interface for ODC

License: Apache License 2.0

Jupyter Notebook 23.73% Python 72.55% C 3.72%

pyodc's Introduction

pyodc

A Python interface to odc for encoding/decoding ODB-2 files.

The package contains two different implementations of the same library:

pyodc is a pure-python encoder and decoder for ODB-2 data, which encodes data from, and decodes it into pandas data frames
codc is an implementation of the same API as pyodc that depends on the ECMWF odc library, and comes with much better performance

Documentation Changelog

Dependencies

Required

Python 3.x

Optional

For codc to work, odc library must be compiled and installed on the system and made available to Python (through the CFFI mechanism) as a shared library. There are multiple ways to make the library visible to CFFI: it can be installed as a system library, the installation prefix can be passed in odc_DIR environment variable, or the library directory can be included in LD_LIBRARY_PATH.

Installation

pip install --user pyodc

Check if the module was installed correctly:

python
>>> import pyodc
>>> import codc # optional

Usage

An introductory Jupyter Notebook with helpful usage examples is provided in the root of this repository:

git clone [email protected]:ecmwf/pyodc.git
cd pyodc
jupyter notebook Introduction.ipynb

Development

Run Unit Tests

To run the unit tests, make sure that the pytest module is installed first:

python -m pytest

Build Documentation

To build the documentation locally, please install the Python dependencies first:

cd docs
pip install -r requirements.txt
make html

The built HTML documentation will be available under the docs/_build/html/index.html path.

License

This software is licensed under the terms of the Apache Licence Version 2.0 which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.

In applying this licence, ECMWF does not waive the privileges and immunities granted to it by virtue of its status as an intergovernmental organisation nor does it submit to any jurisdiction.

pyodc's People

Contributors

Stargazers

Watchers

Forkers

thebaptiste isabella232 dtip tomhodson

pyodc's Issues

pip install breaks codc

pip install codc does not install codc libs correctly, e.g., processed_odc.h was missing from
<python3_install_path>/lib/python3.8/site-packages/codc/

Support new pandas datatypes

Is your feature request related to a problem? Please describe.

Since this code was written, pandas has introduced several new data types that would be a good fit for the data encoded in the ODC file format:

It might be useful to update pyodc to return pandas dataframes that use these new data types where appropriate.

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

Organisation

No response

CHANGELOG.md missing at install via pip

Trying to install pyodc 1.1.0 via pip gives the following error:

$ pip install pyodc==1.1.0
Collecting pyodc==1.1.0
  Using cached pyodc-1.1.0.tar.gz (25 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /home/fabian/anaconda3/envs/test/bin/python /home/fabian/anaconda3/envs/test/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpiv0acpiq
       cwd: /tmp/pip-install-609l767c/pyodc_be4b799dbebc454f9d42773def8ae366
  Complete output (20 lines):
  Traceback (most recent call last):
    File "/home/fabian/anaconda3/envs/test/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 349, in <module>
      main()
    File "/home/fabian/anaconda3/envs/test/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 331, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/home/fabian/anaconda3/envs/test/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 117, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/pip-build-env-ybm4ra2v/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 155, in get_requires_for_build_wheel
      config_settings, requirements=['wheel'])
    File "/tmp/pip-build-env-ybm4ra2v/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 135, in _get_build_requires
      self.run_setup()
    File "/tmp/pip-build-env-ybm4ra2v/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 259, in run_setup
      self).run_setup(setup_script=setup_script)
    File "/tmp/pip-build-env-ybm4ra2v/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 150, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 27, in <module>
      long_description=read("README.md") + read("CHANGELOG.md"),
    File "setup.py", line 10, in read
      return io.open(file_path, encoding="utf-8").read()
  FileNotFoundError: [Errno 2] No such file or directory: 'CHANGELOG.md'
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/e7/bc/8db10ddcadb9dd552baf40543c3b0a74029684c5dbd3880d1e8114b6d4ab/pyodc-1.1.0.tar.gz#sha256=34fd83c5bdeea578a5d0efb1c51b459e6b8ddaf94cd26be2e11c72fb39b68a99 (from https://pypi.org/simple/pyodc/). Command errored out with exit status 1: /home/fabian/anaconda3/envs/test/bin/python /home/fabian/anaconda3/envs/test/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpiv0acpiq Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement pyodc==1.1.0 (from versions: 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.1.0)
ERROR: No matching distribution found for pyodc==1.1.0

I am actually not directly depending on pyodc, but I am using climetlab, whose newest version 0.8.18 depends on pyodc==1.1.0 (i.e. it does not pin a specific version).

Constant string columns truncated to 8 characters

What happened?

After a round trip through encode to odb, decode from odb, strings get truncated to 8 characters.

What are the steps to reproduce the bug?

import pandas as pd
import pyodc as odc
import sys

print("pyodc version: ", odc.__version__)
print("python version: ", sys.version_info)
print("pandas version: ", pd.__version__)


df = pd.DataFrame(dict(col = ["123456789"]*1))
odc.encode_odb(df, "./testcase.odb")
df2 = odc.read_odb("./testcase.odb", single = True)

print(df.col[0], df2.col[0])

On my machine this script prints the following, both in a jupyter notebook and when run as a script in the terminal. In both instances a mamba environment is used and codc is not installed.

pyodc version:  1.3.0
python version:  sys.version_info(major=3, minor=11, micro=3, releaselevel='final', serial=0)
pandas version:  2.0.1
123456789 12345678

The last line should be 123456789 123456789, the 9th character is being cut off.

Version

1.3.0

Platform (OS and architecture)

21.4.0 Darwin Kernel Version 21.4.0: Mon Feb 21 20:35:58 PST 2022; root:xnu-8020.101.4~2/RELEASE_ARM64_T6000 arm64

Relevant log output

No response

Accompanying data

No response

Organisation

ECMWF

PyPI package pyodc-1.0.2 is missing CHANGELOG.rst

The tarball release of pyodc-1.0.2 on PyPI is missing the file CHANGELOG.rst which leads the the following failure to run setup.py.

python3.6 setup.py build -j 16
Traceback (most recent call last):
  File "setup.py", line 27, in <module>
    long_description=read('README.rst') + read('CHANGELOG.rst'),
  File "setup.py", line 10, in read
    return io.open(file_path, encoding='utf-8').read()
FileNotFoundError: [Errno 2] No such file or directory: 'CHANGELOG.rst'

However, the 1.0.2 tarball in the github releases tab does have the CHANGELOG.rst.

ecmwf / pyodc Goto Github PK

pyodc's Introduction

pyodc

Dependencies

Required

Optional

Installation

Usage

Development

Run Unit Tests

Build Documentation

License

pyodc's People

Contributors

Stargazers

Watchers

Forkers

pyodc's Issues

Is your feature request related to a problem? Please describe.

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Organisation

What happened?

What are the steps to reproduce the bug?

Version

Platform (OS and architecture)

Relevant log output

Accompanying data

Organisation

Recommend Projects

Recommend Topics

Recommend Org