Uploading large files seems super slow about hdfs HOT 4 CLOSED

Hi @harshitha-bt,

This is probably due to #84, a really unfortunate performance regression. A fix is on the way.

from hdfs.

(Should be fixed now, in cdda132 and v1.0.4)

from hdfs.

My use case is to download the a multipart message of 700 MB from a rest endpoint and storing it locally and then did a client.CopyToRemote() which took 1 hour for ~700MB of data.
So, i tried using a fileWriter.Write(buf), but this is what i see from file_writer.go ( Write() ).

2017/08/27 20:25:38 offset and len of buffer = 0 706694149
2017/08/27 20:37:01 offset and len of buffer = 134217728 706694149

Even after the fix you mentioned here, it takes 10 mins to upload 1/7th of the file.
What is the best way to upload a file? CopyToRemote() or fileWriter.Write()?

from hdfs.

Any update on this issue? I am seeing performance degradation from cross-zone uploads (also using fileWriter.Write(buf).

from hdfs.