Benchmarks? about datareader HOT 2 OPEN

@kylebarron I haven't done any benchmarks, as I was focused more on correctness. However I believe it is reasonably fast and I routinely use this to process 100's of GBs of data.

I have been involved with maintaining the pandas readers for SAS and Stata (although I was not the primary author for either). I have used the SAS reader much more than the Stata reader, and it is clearly much faster here (in Go) than it is in Python/Pandas. The Stata dta file format is more amenable to vectorized processing, which makes a big difference in Python, so the advantage of using Go might be less for Stata files compared to SAS files (SAS7BDAT is not at all friendly to vectorization).

The column-oriented data structure used here is modeled on Bcolz (https://github.com/Blosc/bcolz), though not interchangeable with it. I believe this is much simpler than Parquet. In any case, I evolved this into a different columnar container called Dstream (https://github.com/kshedden/dstream) which I actively develop and maintain. The container here is mainly for the internal use of the Stata and SAS readers in this package.

Regarding concurrency, the readers work through the SAS/Stata files in chunks, and each chunk has its own backing memory, so you can process one chunk while reading the next. The reading itself is not concurrent (I don't think doing this would help as it is IO-bound).

from datareader.

Thanks for all that information!

from datareader.