Support Parquet as export file format for `COPY TO` about crate HOT 2 OPEN

Can you give some more context on concrete use cases?

COPY FROM is currently designed to do concurrent per shard exports. If you're going to use the parquet files with other tools, I'm not sure if having 1 file per shard is appropriate?

from crate.

Can you give some more context on concrete use cases?

• Long-term storage / Archiving - for efficient long-term storage of data, while still keeping metadata information and being much faster queryable / processed on demand. While this is also possible with gzipped json in a way, one can't easily process them, nor does it keep table schema / data type information.
• Integration with Data Engineering Ecosystems: Simplifies data integration with systems like Databricks / Apache Spark, Hadoop and cloud services (AWS S3, Azure Data Lake).
• Machine Learning: enabling quick access to large datasets for feature selection and model training, reducing the load on the database and preventing system slowdowns.

If you're going to use the parquet files with other tools, I'm not sure if having 1 file per shard is appropriate?

One parquet file per shard would be perfect (one might consider even splitting shards for very large ones). I'd say that it is very common to have a dataset composed of multiple parquet files. One could go a step further and also think about supporting hive-like partitioning.

from crate.