What happened? Test code <div class="highlight highlight-sou

[Bug][Python]: ReadFromCsv with the dtype argument is very slow about beam HOT 3 OPEN

liferoad commented on August 16, 2024

[Bug][Python]: ReadFromCsv with the dtype argument is very slow

from beam.

Comments (3)

liferoad commented on August 16, 2024

One workaround is to convert the value using your own function, for example,

with beam.Pipeline() as pipeline:
    (
        pipeline
        | beam.io.ReadFromCsv("input.csv", dtype=None)
        | beam.Map(lambda x: [str(t) for t in x])
        | beam.Map(print)
    )

from beam.

sorensenjs commented on August 16, 2024

That workaround code example doesn't work, because ReadFromCsv and WriteToCsv both produce/require schemas.

Even if you were to use a ._asdict() like so

( pipeline | beam.io.ReadFromCsv('/tmp/input.csv', dtype=str)
      | beam.Map(lambda x: x._asdict())
      | beam.Map(print)

Conversion to str element by element will result in 'None' values instead of empty strings. Values interpreted as floating point may loose precision.

{'a': 'text', 'b': 1, 'c': 21, 'd': 5945023, 'e': 376974, 'f': 0, 'g': 0, 'h': 0, 'i': 1, 'j': 2, 'k': 0, 'l': 4, 'm': None, 'n': None, 'o': None, 'p': None, 'q': None, 'r': None}
Elapsed time  0:00:00.878320
{'a': 'text', 'b': '1', 'c': '21', 'd': '5945023', 'e': '376974', 'f': '0', 'g': '0', 'h': '0', 'i': '1', 'j': '2', 'k': '0', 'l': '4', 'm': None, 'n': None, 'o': None, 'p': None, 'q': None, 'r': None}
Elapsed time  0:00:24.243182

from beam.

liferoad commented on August 16, 2024

My example just shows that you can do the conversation without using dtype=str. If you need to keep the schemas, you could do something like:

with beam.Pipeline() as pipeline:
    (
        pipeline
        | beam.io.ReadFromCsv("input.csv", dtype=None)
        | beam.Map(lambda x: beam.Row(a=x.a, b=str(x.b)))
        | beam.io.WriteToCsv("output1.csv")
    )

from beam.

Recommend Projects