Reduce shuffle write size in DistributedLDA about mammoth HOT 2 CLOSED

Accumulators seem promising. The documentation only mentions accumulators for single values, but perhaps they can be extended to Matrices.

from mammoth.

Accumulators suffer from the same problem as they are implemented internally as a reduce operation.

It turns out that this is intended behavior of Spark. Any reduce operation is a 'pull' operation not a 'push' operation. As a consequence this means that the full shuffle write should fit in the cluster (preferably in memory, but a spill-to-disk mechanic does exist in the newest versions of spark). As this grows with the number of partitions (and thus the data set), this can be problematic if the full data set does not fit.
For more info check section 3.2.2 of Optimizing Shuffle Performance in Spark.

from mammoth.