clintpgeorge / tm Goto Github PK

Test the hyperparameter estimation and Gibbs sampling algorithms ( collapsed and full ) on a couple of real world data sets and benchmark them

There was a couple of R system crashes, when I ran the LDA Gibbs sampling algorithm for a large set of documents ( test_lda_c.R ). This happened at the function call that handles the C++ - R transfer of objects such as betas (K x V x G matrix), and thetas (K x D x G matrix), and Z (N x G), where K is the number of topics, V is the vocabulary size, D is the number of documents in the corpus, G is the number of saved MCMC iterations, and N is the number of word instances in the corpus.

One solution would be to keep only z values ( Z matrix ) and computing the betas and thetas on demand, i.e., when we do the computation of likelihood ratios. For example, the function

compute_thetas <- function(did, Z, K, D, base.alpha.v)

in utils.R computes thetas from the stored Z matrix.

Note: This could be an issue with the way RcppArmadillo handles _cube_ data structure and Rcpp transfer it to R environment as an _array_ data structure.

Frequency: rare

Plot the respective posteriors from the Markov chain and see if it's actually optimizing the posterior