ktw5691 / psychtm Goto Github PK

How can models fitted with/without topics be compared?

For instance, in the following code:

with_category = gibbs_sldax(log(HoursActual) ~ log(HoursEstimate)+Category+ProjectCode, data = cl_info,
                         m = 450, burn = 300,
                         docs = docs_vocab$documents,
                         V = vocab_len,
                         K = 10, model = "sldax", display_progress = TRUE)

what would be the impact on goodness of fit, from just fitting:
log(HoursActual) ~ log(HoursEstimate)+Category+ProjectCode?

That is, how can the impact of topics be separated out from the other covariates?

A quality of fit metric, such as AIC, would be useful output from summary, assuming that upstream functions generate the necessary information.

Currently, gibbs_sldax() checks for missing data in ANY variable in the data frame supplied to the data argument. It will stop and warn the user about missing values even if the variables being used for modeling have no missing data.

Being clueless, I typed in the code from the help page, i.e., iter=5, and it worked, or rather it produced numbers without any complaint.

Then I noticed that the output from summary contained a chain value (for my code, 1), so I tried that. It produced different output.

Then I noticed that in the example, the call to gibbs_mlr included the argument m=5. My call to gibbs_sldax has m=450, and waic_all produces different output when given 450.

The second parameter of waic_all, l_pred, is an 'iter' x D matrix.

The value of iter can be obtained from the argument passed. Why is the first parameter, iter, needed?

The help is not helpful with regard to legitimate values of iter. Not giving one does produce an error message.

bigmemory R package?

Bad initialization of regression coefficients can start the algorithm with infinite values from which it never recovers.

Having to do stop-word removal and stemming is a hassle.

Be helpful, like the textProcessor function in the stm package, do it automatically (by default).

First release:

• usethis::use_cran_comments()
• Update (aspirational) install instructions in README
• Proofread Title: and Description:
• Check that all exported functions have @return and @examples
• Check that Authors@R: includes a copyright holder (role 'cph')
• Check licensing of included files
• Review https://github.com/DavisVaughan/extrachecks

Prepare for release:

• devtools::build_readme()
• urlchecker::url_check()
• devtools::check(remote = TRUE, manual = TRUE)
• devtools::check_win_devel()
• rhub::check_for_cran()
• rhub::check(platform = 'ubuntu-rchk')
• rhub::check_with_sanitizers()
• Draft blog post

Submit to CRAN:

• usethis::use_version('minor')
• devtools::submit_cran()
• Approve email

Wait for CRAN...

• Accepted 🎉
• usethis::use_github_release()
• usethis::use_dev_version()
• Finish blog post
• Tweet
• Add link to blog post in pkgdown news menu

Provide vignette to illustrate package use.

• Need to use one of topic assignment samples, topic proportion samples, or vocabulary samples.
• Keep the permutation indicators and use to realign all relevant model parameters (regression coefficients, topic proportions, topic-vocabulary distributions).

Package checking using the goodpractice R package (which uses the cyclocomp package) flagged the gibbs_*() functions as having high cyclometric complexity. This could be reduced by writing more helper functions to take up tasks in the gibbs_*() functions.

See https://adv-r.hadley.nz/s4.html#s4-classes Section 15.3.5

With verbose = TRUE and model = "lda", output:
1 eta: 0
500 zbar d1: 0 6.9351e-310 nan
1000 zbar d1: 0 6.9351e-310 nan
...

Should be able to adapt code for sLDA sampler

The check in prep_docs: if (any(is.na(data))) stop("Cannot handle missing values in 'data'") is too broad. Only the columns specified in col need to be checked.

Having to remove NA from columns that are not referenced by the called psychtm function is a pain.

Currently, the default contrasts computed in post_regression use equal weighting for all reference topics being tested against for each topic effect. It would be nice to implement Piepel-type (1986) contrasts to take into account possible range restriction in the empirical topic proportions as well as their sampling variability.

# Example usage
post_regression(sldax_fit, ctype = c("equal", "piepel"))

Expand unit tests: next target is 25%.

• See tinytest for C++ code unit tests.

The message:
"Each document (row) in 'docs' must contain
at least 2 positive integers"
is not helpful. Why not say: "The text must contain at least two words."

Even better, why not automatically remove rows that don't contain some minimum number of words?

It took me 30 minutes to figure out which columns I needed to remove in the original data. For code+data see: https://shape-of-code.com/2022/01/23/including-natural-language-text-topics-in-a-regression-model/

Suggested by @alexbrodersen

A Bayesian $R^2$ metrics for supervised models with a continuous outcome (MLR, SLDA, SLDAX) would be helpful.

• This is straightforward for MLR.
• For SLDA and SLDAX, this requires computing empirical topic proportions (or keeping the topic assignments and then computing after sampling, which can be memory expensive).
  • Alternatively, an approximate $R^2$ could be computed using $\hat{\Theta}$ instead.

# Example usage
bayes_r2(sldax_fit)

• Rcpp v.1.0.8 adds support for alternative degrees of full functionality from Rcpp:

  • inst/include/Rcpp/Rcpp: Added as new entry point
  • inst/include/Rcpp/Light: Added as lighter-weight entry point
  • inst/include/Rcpp/Lighter: Idem
  • inst/include/Rcpp/Lightest: Idem

• See changed files for specific differences.

Set up a website for the package using packagedown.

• Integrate through GitHub Pages?
• Or through personal site?

If they mismatch:

Error in .gibbs_sldax_cpp(docs, V, m, burn, thin, K, model, y, x, mu0,  : join_rows() / join_horiz(): number of rows must be the same

Currently they are calculated based on the full model, not the collapsed model

I think this would be doable. The posterior mean estimates should just be weighted sums that could be updated at each non-thinned iteration after burn-in.