Rss shuffle data size is much larger than external shuffle service about remoteshuffleservice HOT 6 CLOSED

Hi @YutingWang98, this is interesting finding! For the size 720 TB/370 TB, is this disk file size, or the stats shown in Spark UI?

from remoteshuffleservice.

Hi @hiboyang, thank you for the reply! It is the stats from spark UI stage tab ('Shuffle Write Size / Records' value). Also, I just did more tests on a same job and here are my findings:

Remote shuffle service

• rss without compression: 5.7GB
• rss with compression (lz4): 3.1GB
• made changes in rss and allow it to use zstd:
  • compression level=1 2.4 GB
  • compression level=7 2031 MB

External shuffle service

• default setting (zstd compression, compression level=1): 1927 MB
• without compression (spark.shuffle.service.enabled=false): 5.6GB
• other compressions methods
  • spark.io.compression.codec=lz4: 2.7 GB
  • spark.io.compression.codec=lzf 2.7GB
  • spark.io.compression.codec=snappy 2.7 GB

So, I think switching to zstd might be helpful.

from remoteshuffleservice.

Nice to know zstd has better compression ratio. I created a PR to support zstd in RSS: https://github.com/uber/RemoteShuffleService/pull/91/files

from remoteshuffleservice.

Thank you! WIll test our job with this new change. Also I think in spark compression, the zstd compression level 'spark.io.compression.zstd.level' is set to 1 as default. But I saw you are using level 3 as default. Is there any specific reason for it?

from remoteshuffleservice.

No specific reason :) I changed it to level 1 in the PR, but forgot to reply the comment here.

from remoteshuffleservice.

I saw your changes in the pull request! Thanks so much.

from remoteshuffleservice.