Why Spark produces performance data based on csv dataset about db-benchmark HOT 8 CLOSED

It may be easier if you just run db-benchmark script having data source replaced and compare timings.
Caching data in memory，the performance of csv and parquet is indeed similar in db-benchmark's scense.

from db-benchmark.

Do you mean that Spark is not dealing well with CSV data?
It loads the data into memory so CSV is just a source for data. Later on spark operates on an in-memory object and not the CSV.

from db-benchmark.

Do you mean that Spark is not dealing well with CSV data? It loads the data into memory so CSV is just a source for data. Later on spark operates on an in-memory object and not the CSV.
Spark has specifically optimized parquet files. Take q1 of 5G data groupby operation as an example，spark processing csv dataset takes 8.42s on average. but spark processing parquet dataset takes 555ms on average.
csv data:

parquet data:

from db-benchmark.

Thanks for investigation. You should try to cache data in memory before running first query, as we do in benchmark script. As of now your code is probably including time to load data from csv in the time of query.

from db-benchmark.

Thanks for investigation. You should try to cache data in memory before running first query, as we do in benchmark script. As of now your code is probably including time to load data from csv in the time of query.
Refer to the suggestions above，I try to cache data in memory before running first query，but the result is basically the same.

from db-benchmark.

I don't think that your code caches data in memory. Spark interface is lazy and AFAIR calling persist alone does not force computation.

from db-benchmark.

I don't think that your code caches data in memory. Spark interface is lazy and AFAIR calling persist alone does not force computation.

See my code in the attachment, you can take a look at it as a whole.
spark_csv_vs_parquet.ipynb.zip

from db-benchmark.

It may be easier if you just run db-benchmark script having data source replaced and compare timings.

from db-benchmark.