Usage of shuffle in tarp about tarp HOT 1 CLOSED

The data will be shuffled in a shuffle buffer of size 5000 and then split into shards of 10000. This means that most of the samples that were together in the original tar file will still be close together in the same shards in the output. Since tarp does all processing sequentially, this is really all it can do.

You have two options:

• greatly increase the size of the shuffle buffer; make it 10x the size of the output shards and you'll get good mixing across shards
• first, split the original file into many small shards and then shuffle those shards and do a parallel cat with tarp from multiple shards simultaneously

from tarp.