sdpython / pandas-streaming Goto Github PK

View Code? Open in Web Editor NEW

28.0 7.0 9.0 850 KB

Streaming API for pandas applied to big datasets

Home Page: https://sdpython.github.io/doc/pandas-streaming/dev/

License: MIT License

Python 100.00%

python3 pandas numpy streaming-data streaming-data-processing

pandas-streaming's Introduction

pandas-streaming: streaming API over pandas

https://dl.circleci.com/status-badge/img/gh/sdpython/pandas-streaming/tree/main.svg?style=svg

https://dev.azure.com/xavierdupre3/pandas_streaming/_apis/build/status/sdpython.pandas_streaming

https://codecov.io/gh/sdpython/pandas-streaming/branch/main/graph/badge.svg?token=0caHX1rhr8

pandas-streaming aims at processing big files with pandas, too big to hold in memory, too small to be parallelized with a significant gain. The module replicates a subset of pandas API and implements other functionalities for machine learning.

from pandas_streaming.df import StreamingDataFrame
sdf = StreamingDataFrame.read_csv("filename", sep="\t", encoding="utf-8")

for df in sdf:
    # process this chunk of data
    # df is a dataframe
    print(df)

The module can also stream an existing dataframe.

import pandas
df = pandas.DataFrame([dict(cf=0, cint=0, cstr="0"),
                       dict(cf=1, cint=1, cstr="1"),
                       dict(cf=3, cint=3, cstr="3")])

from pandas_streaming.df import StreamingDataFrame
sdf = StreamingDataFrame.read_df(df)

for df in sdf:
    # process this chunk of data
    # df is a dataframe
    print(df)

It contains other helpers to split datasets into train and test with some weird constraints.

pandas-streaming's People

Contributors

Stargazers

Watchers

Forkers

revirevy pandinosaurus chanpatrick hamzamoudnibeconocom webclinic017 king9sun mattburnham erdal-pb ownermz

pandas-streaming's Issues

implements fillna

Add sdist to PyPI releases

Hi! 👋

Thanks for maintaining this package. I have added pandas-streaming to conda-forge but it was a bit tricky because the PyPI release doesn't contain an sdist.

Could you maybe add an sdist for future releases? That would help me a lot. Thanks! 😃

fix pandas_streaming for pandas 0.23.1

implement read_json

for json organized as rows or not (extension compare to pandas.read_json)

method train_test_split for out-of-memory datasets

implement concat for axis=0,1

Excited for your project

bonjour - I came across this project working with ElasticSearch + Pandas. Will be looking forward to your work and will hope to contribute changes too

add random_state parameter to splitting functions

Support of streaming data sources

Like receiving io streams and quantization criterion, Kafka and other remote queues in a similar way like Spark streaming works.

Use tqdm to display progress bar

add pandas_groupby_nan from pyensae

groupby including nan values

implement BJKST

add method sample, resevoir sampling

add groupby_streaming

to see the first step of the aggregation

head is very slow

implements parameter lines, flatten for read_json

Unit tests failing with pandas 1.1.0.

add method add_column

Missing packages: scikit-learn and ijson

Pip doesn't install all required packages. It seems that scikit-learn and ijson are missing.

The steps to demonstrate that scikit-learn is missing:

python -m venv venv
source venv/bin/activate
pip install pandas-streaming
python -c "import pandas_streaming.df"

And, the output:

ModuleNotFoundError: No module named 'sklearn'

The steps to demonstrate that ijson is missing:

pip install scikit-learn
python -c "import pandas_streaming.df"