Optimize data loading. about desktop HOT 6 CLOSED

SQLite is much more than just reading a table. There could be some better binary format for this.

from desktop.

Can we hook the CSV reader so that it emits rows to another parallel thread and stores to an sqlite DB (or other fast to read format)? This could happen in the background while a query is doing a full scan.

It makes more sense to me to do this in Easy Diffix, after schema detection.

Not sure how much we would gain by this.

If we don't plan to use column indexes, I don't have high hopes it would be much faster.
I would prefer if we were able to read compressed data. I also don't want to have to worry about cache storage and garbage collection.

from desktop.

I would prefer if we were able to read compressed data.

I gave this a try, but got no significant performance gain.

I don't think we should do data caching (adds too many headaches). We could try to use separate threads for data loading and aggregation, maybe that makes an impact.

from desktop.

I had tried to use parallelization via PLINQ and such but it didn't make a difference (while doing the immutable data structure optimization).

Performance is quite good now, we just need to figure out how to optimize the DISTINCT count. I noticed that it is highly dependent on the data structure. It scales terribly with number of buckets I think.

from desktop.

OK, there is a separate issue in reference for that already, so this seems obsolete now.

from desktop.

It scales terribly with number of buckets I think.

My impression was that it scales badly with the cardinality of the input argument.

from desktop.