Blocks read inconsistent about firestorm HOT 6 CLOSED

"Blocks read inconsistent" means can't read expected blocks which are sent to shuffle server. It may be caused by write problem. For storage type, please use MEMORY_HDFS instead of HDFS.

from firestorm.

MEMORY_LOCALFILE has the same problem.

from firestorm.

用local_hdfs模式，你会发现，本地存的和hdfs上存的数据不一样，然后报找不到文件的错误。

from firestorm.

MEMORY_LOCALFILE has the same problem.

What's the configuration of shuffle server?

from firestorm.

MEMORY_LOCALFILE has the same problem.

What's the configuration of shuffle server?

rss.rpc.server.port 19999
rss.jetty.http.port 19998
rss.storage.basePath /HDATA/1/rssdata,/HDATA/2/rssdata,/HDATA/3/rssdata,/HDATA/4/rssdata,/HDATA/5/rssdata,/HDATA/6/rssdata
#rss.storage.type MEMORY_LOCALFILE
rss.storage.type MEMORY_LOCALFILE_HDFS
rss.coordinator.quorum coordinator1:19999, coordinator2:19999
rss.server.buffer.capacity 40gb
rss.server.buffer.spill.threshold 22gb
rss.server.partition.buffer.size 150mb
rss.server.read.buffer.capacity 20gb
rss.server.flush.thread.alive 50
rss.server.flush.threadPool.size 100

rss.server.hdfs.base.path hdfs://ns/tmp/rss/hdfs_base_path

# multistorage config
rss.server.multistorage.enable true
#rss.server.uploader.enable false
#rss.server.uploader.base.path hdfs://ns/tmp/rss/uploader_base_path
#rss.server.uploader.thread.number 32
# rss.server.disk.capacity 1011550697553

from firestorm.

Client should keep the storage type consistent with server.

from firestorm.