The goal of this workflow is to validate a small Bayesian model using simulation-based calibration (SBC; Cook, Gelman, and Rubin 2006; Talts et al. 2020). We simulate multiple datasets from the model and fit the model on each dataset. For each model fit, we determine if the 50% credible interval of the regression coefficient beta contains the true value of beta used to generate the data. If we implemented the model correctly, roughly 50% of the models should recapture the true beta in 50% credible intervals.

y_i ~ iid Normal(alpha + x_i * beta, sigma^2)
alpha ~ Normal(0, 1)
beta ~ Normal(0, 1)
sigma ~ HalfCauchy(0, 1)

The targets R package manages the workflow. It automatically skips steps of the pipeline when the results are already up to date, which is critical for Bayesian data analysis because it usually takes a long time to run Markov chain Monte Carlo. It also helps users understand and communicate this work with tools like the interactive dependency graph below.

library(targets)
tar_visnetwork()

The files in this example are organized as follows.

├── run.sh
├── run.R
├── _targets.R
├── _targets/
├── sge.tmpl
├── R
│   ├── functions.R
│   └── utils.R
├── stan
│   └── model.stan
└── report.Rmd

This project has an RStudio Cloud workspace that lets you try out the example code in the cloud with only a web browser and an internet connection. Unfortunately, as explained here, rstan cannot currently compile models in RStudio Cloud because it hits the 1 GB memory limit. The Stan maintainers will likely fix this at some point.

1. If you are running locally instead of this RStudio cloud workspace
   1. Install the targets package, as well as the packages listed in the tar_option_set() call in _targets.R.
   2. Download the files in this repository, either through Git or through this link.
2. Run the targets pipeline by either running run.R or run.sh. (The latter is for Unix-like systems only). This computation could take a while.
3. View the validation results in the output report.html file.
4. Make changes to the R code or Stan model, rerun the pipeline, and watch targets skip steps that are already up to date.

This computation is currently downsized for pedagogical purposes. To scale it up, open the _targets.R script and increase the number of simulations (the number inside seq_len() in the index target).

You can run this project locally on your laptop or remotely on a cluster. You have several choices, and they each require modifications to run.R and _targets.R.

The stantargets R package is an extension to targets and cmdstanr for Bayesian data analysis. stantargets makes it super easy to set up useful scalable Stan pipelines that automatically parallelize the computation and skip expensive steps when the results are already up to date. Minimal custom code is required, and there is no need to manually configure branching, so usage is much easier than targets alone. stantargets can access all of cmdstanr’s major algorithms (MCMC, variational Bayes, and optimization) and it supports both single-fit workflows and multi-rep simulation studies.

stantargets condenses the workflow in this repo down to this simple pipeline statement without loss of technical sophistication or computing power. The former requires users to think carefully about dynamic branching and file tracking, and the latter uses domain knowledge to abstract away these intimidating concepts.

Cook, Samantha R., Andrew Gelman, and Donald B. Rubin. 2006. “Validation of Software for Bayesian Models Using Posterior Quantiles.” Journal of Computational and Graphical Statistics 15 (3). [American Statistical Association, Taylor & Francis, Ltd., Institute of Mathematical Statistics, Interface Foundation of America]: 675–92. http://www.jstor.org/stable/27594203.

Talts, Sean, Michael Betancourt, Daniel Simpson, Aki Vehtari, and Andrew Gelman. 2020. “Validating Bayesian Inference Algorithms with Simulation-Based Calibration.”