How to run simulation-based calibration (SBC)

How to run simulation-based calibration (SBC)#

Similar to expected coverage, simulation-based calibration (SBC) provides a simple and interpretable tool to diagnose issues in the posterior. It also requires relatively few additional simulations (~200) and it does not rely on any additional hyperparameters (as TARP would) or additional neural network training.

SBC allows you to evaluate whether individual marginals of the posterior are, on average across many observations (prior predictive samples) too narrow, too wide, or skewed.

You can run SBC with the sbi toolbox as shown below:

Main syntax#

from sbi.diagnostics import run_sbc
from sbi.analysis.plot import sbc_rank_plot

# Obtain your `posterior_estimator` with NPE, NLE, NRE.
posterior = inference.build_posterior()

num_sbc_samples = 200  # choose a number of sbc runs, should be ~100s
prior_samples = prior.sample((num_sbc_samples,))
prior_predictives = simulate(prior_samples)

# run SBC: for each inference we draw 1000 posterior samples.
num_posterior_samples = 1_000
ranks, dap_samples = run_sbc(
    prior_samples,
    prior_predictives,
    posterior,
    num_posterior_samples=num_posterior_samples,
    use_batched_sampling=False,  # `True` can give speed-ups, but can cause memory issues.
)
fig, ax = sbc_rank_plot(
    ranks,
    num_posterior_samples,
    num_bins=20,
    figsize=(5, 3),
)

The only difference to running expected coverage is that we did not pass run_sbc(..., reduce_fns=...) and we visualize it differently by not passing sbc_rank_plot(..., plot_type="cdf")

This will return a figure such as the following:

This plots as many plots as there are parameters. For each of the parameters you can interpret the the shape of the red bars as follows:

  • if they are all contained within the gray region, then we cannot reject the null-hypothesis that the posterior marginals are well-calibrated.

  • If the red bars are higher on the left (i.e. they are sloped downwards), then the posterior estimate is skewed to the right

  • If the red bars are higher on the right (i.e. they are sloped upwards), then the posterior estimate is skewed to the left

  • If the red bars are U-shaped, then our posterior estimate is too narrow in that parameter dimension

  • If the red bars are bell-shaped, then our posterior estimate is too wide in that parameter dimension

Example#

For a detailed example and further explanation, see this tutorial.

Citation#

@article{cook2006validation,
  title={Validation of software for Bayesian models using posterior quantiles},
  author={Cook, Samantha R and Gelman, Andrew and Rubin, Donald B},
  journal={Journal of Computational and Graphical Statistics},
  volume={15},
  number={3},
  pages={675--692},
  year={2006},
  publisher={Taylor \& Francis}
}

@article{talts2018validating,
  title={Validating Bayesian inference algorithms with simulation-based calibration},
  author={Talts, Sean and Betancourt, Michael and Simpson, Daniel and Vehtari, Aki and Gelman, Andrew},
  journal={arXiv preprint arXiv:1804.06788},
  year={2018}
}
Navigation