[Enhancement] download module about cotk HOT 4 CLOSED

• dataset, wordvec ... are called as resources
• every resources has a config
  • resources name (unique)
  • resources type (identify postprocess)
  • resources hashtag
  • resources link
  • other things download module needs (as least as possible)
• download pipeline
  • given resources name
  • find config
  • if not cached, download from link. Process after download (find by type), save cache.
  • if cached, read from local disk.
  • check hashtag
  • process before read (find by type) (*)
  • provide to other module
• import benchmark from local
  • given resources name and local path
  • check the hash tag
  • import to cache
  • continue step (*)
• import temporary resources from local
  • local path and type
  • continue step(*)

from cotk.

TODO:

• import temporary resources from local
• fill the dataset config and type

from cotk.

https://github.com/thu-coai/contk/blob/e5094bff5491216245f311df376d64e9636f6f64/contk/dataloader/language_generation.py#L115-L140

• file_path change to file_id:

  • if file_id is like "resources://RESOURCE_ID", find RESOURCE_ID in configs. (Check the config type == MSCOCO)
  • if file_id starts with "http" or "https", download and cache it. (set resources type = MSCOCO)
  • otherwise, file_id is a local path. Don't cache it. (set resources type = MSCOCO)
  • change other dataloader & wordvector

• dataset download path:

  • http://coai.cs.tsinghua.edu.cn/file/mscoco.zip
  • http://coai.cs.tsinghua.edu.cn/file/opensubtitles.zip
  • http://coai.cs.tsinghua.edu.cn/file/ubuntu_dataset.zip

from cotk.

Done

from cotk.