load_dataset doesn't support list column about datasets HOT 1 OPEN

I encountered this same issue when loading a customized dataset for ORPO training, in which there were three columns and two of them were lists.
I debugged and found that it might be caused by the type-infer mechanism and because in some chunks one of the columns is always an empty list ([]), it was regarded as list<item: null>, however in some other chunk it was list<item: string>. This triggered a TypeError running the function table_cast().

I temporarily fixed this by re-dumping the file into a regular JSON format instead of lines of JSON dict. I didn't dig deeper for the lack of knowledge and programming ability but I do hope some developer of this repo will find and fix it.

from datasets.