[Question] Using/adding a local dataset about datasets HOT 5 CLOSED

Hi @MaveriQ , although what I am doing is to commit a new dataset, but I think looking at imdb script might help.
You may want to use dl_manager.download_custom, give it a url(arbitrary string), a custom_download(arbitrary function) and return a path, and finally use _get sample to fetch a sample.

from datasets.

The download manager supports local directories. You can specify a local directory instead of a url and it should work.

from datasets.

Hi @zphang,

So you can just give the local path to a dataset script file and it should work.

Here is an example:

• you can download one of the scripts in the datasets folder of the present repo (or clone the repo)
• then you can load it with load_dataset('PATH/TO/YOUR/LOCAL/SCRIPT.py')

Does it make sense?

from datasets.

Could you give a more concrete example, please?

I looked up wikitext dataset script from the repo. Should I just overwrite the data_file on line 98 to point to the local dataset directory? Would it work for different configurations of wikitext (wikitext2, wikitext103 etc.)?

Or maybe we can use DownloadManager to specify local dataset location? In that case, where do we use DownloadManager instance?

Thanks

from datasets.

Closing this one.
Feel free to re-open if you have other questions :)

from datasets.