This would also make it possible to continue working with the kaggle data instead of needing two separate directories.

most preferably with #13 in place due to storage requirements

• use .parquet files
• convert to a lower precision dtype, e.g. float32 and int16.
• drop some unnecessary columns

This brought a CSV file of 170MB back to 52MB; ~30%. 50GB dataset (all pairs) then becomes 15GB!

This creates better readable csv files and displays them better on the kaggle page.

can't find docs

https://stackoverflow.com/questions/55934733/documentation-for-kaggle-api-within-python

Didn't get further than this:

import kaggle
kaggle.api.authenticate()

Right now using CLI: kaggle datasets version -p . -m "<NOTE>"

e.g.:

    "data": [
        {
            "description": null,
            "name": "ADA-BTC.csv",
            "totalBytes": 126269896,
            "columns": [
                {
                    "name": "open_time",
                    "description": null,
                    "type": "Uuid"
                },
                {
                    "name": "open",
                    "description": null,
                    "type": "Uuid"
                },

etc.

Currency pairs that do not have candlesticks on the day of running the script have to be redownloaded again, due to this check: https://github.com/gosuto-ai/candlestick_scraper/blob/b18eadce3f42f9f764dd4cd44bf156817b8d7140/main.py#L62

This applies to maybe 30 out of the ~200 BTC pairs.

Pandas to_parquet() method uses snappy compression by default. You can get significantly better compression (20% lower file size or better) and keep good decompression speed by passing compression=zstd when saving to parquet.
It's worth noting that zstd allows many compression levels and I'm not sure if pandas automatically chooses the highest compression level, importing the df into a duckdb table and saving the parquet from there specifying zstd as compression could result in lower-sized parquet files

current setup has unnecessary io bottlebeck

https://pip.pypa.io/en/stable/user_guide/#requirements-files

if no candlesticks are available for this day yet

can be solved by comparing last_datetime at beginning and at end of the loop

Add the ability to catch other connection errors such as time out and connection reset error.
Just made a pull request

Pairs that didn't trade yet on the day of running the script are checked anyway and therefore generate the last known timestamp twice.