Split dataset between workers about datasets HOT 3 CLOSED

Is there any update on this issue? Will there be some sort of multiprocess support in the future?

If not, will there be some guides on how to utilize multiprocessing in the best way using this package?

from datasets.

The current plan is to let the user select the shards they want with: #121

Not sure when I'll have time to implement this in a close future though.

from datasets.

For update, the Split API has been updated to be more performant and only read required shards, rather than the whole dataset.
This should significantly increase performances.

For automatic multiprocessing, I'm closing this issue in favor of #1426

from datasets.