ismailhammounou / db2ixf Goto Github PK

pyarrow_record_batches should be a helper function so users can use it to maybe add new columns internally to the parsed data or maybe to parse new formats ... etc

Separate jobs and maybe try use multiple environments or workflows ?

Hi @ismailhammounou,

I was trying to use your tool to convert to CSV some IXF files and I've got The column {col_name} has unknown data type error for VARGRAPHIC data type.

Considering that...

I've forked your repo and did some changes in order to get the VARGRAPHIC '464' data type being parsed as a string. I have:

• Added a new collector;
• Added a 'VarGraphicLengthException' class;
• Added the references for these additions;

I have also:

• Applied .strip() for time, date and timestamp while collecting the column data because for some reason I had IXF files with spaces after the date/time/timestamp;
• Applied a fallback format for the timestamp collector while converting. The reason is because I had an IXF file that wasn't matching the defined timestamp format;

You can see the mentioned changes through the following commit: rckmath@e60aba5

That's my first hands on with Python and also in contributing to open source projects, and that's why I'm not directly opening a pull request into your repository.

I will appreciate any feedback.

EOL of python 3.7 on 30/06/2023

As a data scientist/engineer

I want to be able to parse IBM exchange format and output the result in either json, csv or a parquet

in order to be analysed by other systems like spark for exemple

blob collector for BLOB data type does not take into consideration the code page into account and it affects the encoding/decoding

As a user
I want more documentation about the CLI
In order to use it easily

Package vercion 0.7.0 currently returns TypeError: 'type' object is not subscriptable at usage.

# coding=utf-8
import pathlib
from db2ixf.ixf import IXFParser

path = pathlib.Path('Path/to/IXF/FILE/XXX.IXF')
with open(path, mode='rb') as f:
    parser = IXFParser(f)
    output_path = pathlib.Path('Path/To/Output/YYY.csv')
    with open(output_path, mode='w', encoding='utf-8') as output_file:
        parser.to_csv(output_file, sep='#')

returns

Traceback (most recent call last):
  File "test.py", line 3, in <module>
    from db2ixf.ixf import IXFParser
  File "db2ixf\__init__.py", line 94, in <module>
    from db2ixf.ixf import IXFParser
  File "db2ixf\ixf.py", line 11, in <module>
    from db2ixf.collectors import (collect_bigint,
  File "db2ixf\collectors.py", line 9, in <module>
    from db2ixf.helpers import get_ccsid_from_column
  File "db2ixf\helpers.py", line 10, in <module>
    def get_pyarrow_schema(cols: list[dict]) -> dict[str, object]:
TypeError: 'type' object is not subscriptable

If you want to maintain compatibility with Python 3.8, you need to change

def get_pyarrow_schema(cols: List[dict]) -> Dict[str, object]:

and

def get_pandas_schema(cols: List[dict]) -> Dict[str, object]:

and

def merge_dicts(dicts: List[dict]) -> Dict[str, list]:

Thank you for the package!