Arrow Invalid error while running this training script for FUNSD training script using huggingface trainer about transformers-tutorials HOT 5 CLOSED

problems solved in #24 (comment)

from transformers-tutorials.

same error when running this link

from transformers-tutorials.

It seems like the PR fixed the problem in most cases, but when the data is a list of lists (and not a np array), pyarrow is still incapable of dealing with it as is. A quick and ugly fix for the moment can be to cast the lists as a np array.

Ugly fix (waiting for a better option). Change datasets/arrow_writer.py as such:

  try:
      if isinstance(type, _ArrayXDExtensionType):
          self.data = np.array(self.data)     # Here is the line to add
          if isinstance(self.data, np.ndarray):
              storage = numpy_to_pyarrow_listarray(self.data, type=type.value_type)
          else:
              storage = pa.array(self.data, type.storage_dtype)
          out = pa.ExtensionArray.from_storage(type, storage)

from transformers-tutorials.

See #36

from transformers-tutorials.

The issue linked above should solve your problem. Therefore, closing this issue.

from transformers-tutorials.