From PR 43 about cramming HOT 5 CLOSED

I originally faced the same error msg with @euclaise , and after the proposed change is applied, I instead face this error message.

The work-around I made to this problem is to change the line from

tokenized_dataset = datasets.load_dataset(cfg_data.hf_location, "train", streaming=cfg_data.streaming, cache_dir=data_path)["train"]

to

tokenized_dataset = datasets.load_dataset(cfg_data.hf_location, "default", streaming=cfg_data.streaming, cache_dir=data_path)["train"]

And it solves the problem, however I'm not familiar with the datasets package so I'm not sure if it is the right fix.

from cramming.

This fix is now included in commit 2875a3b thanks!

from cramming.

Is this still an issue?

from cramming.

Problem might be related to differences in versions of the datasets package. The fix from the PR is only necessary for newer releases, and a problem for older ones.

from cramming.

I originally faced the same error msg with @euclaise , and after the proposed change is applied, I instead face this error message.

The work-around I made to this problem is to change the line from

tokenized_dataset = datasets.load_dataset(cfg_data.hf_location, "train", streaming=cfg_data.streaming, cache_dir=data_path)["train"]

to

tokenized_dataset = datasets.load_dataset(cfg_data.hf_location, "default", streaming=cfg_data.streaming, cache_dir=data_path)["train"]

And it solves the problem, however I'm not familiar with the datasets package so I'm not sure if it is the right fix.

This worked for me! For anyone interested, I am running Python 3.10, datasets 2.18.0, Ubuntu 22.0.

from cramming.