rjagerman / mammoth Goto Github PK

The shuffle write size grows with the number of topics (K), the number of vocabulary terms (V) and the number of documents. For relatively small topic models (K=50, V=100,000) the shuffle write size is already larger than the input size (e.g. we put in 40GB of data and this results in a 50GB shuffle write).

Running LDA on a data set larger than the cluster's storage size will likely result in failures due to insufficient space for the shuffle writes.

Regardless of parameter n, the LDAModel's printTopics function only prints out 10 terms per topic.

The current local LDA solver runs into issues with large amounts of data due to spark heartbeat timeouts.

Currently models are summed by returning and adding the β matrices element-wise. This only really works for small β matrices. To scale this up we would want to emit all the elements (j,k) of β, and use a map-reduce style summation.