sbi.diagnostics.run_tarp

Navigation

sbi.diagnostics.run_tarp#

run_tarp(thetas, xs, posterior, references=None, num_posterior_samples=1000, num_workers=1, show_progress_bar=True, distance=<function l2>, num_bins=None, z_score_theta=True, use_batched_sampling=True)[source]#

Estimates coverage of samples given true values thetas with the TARP method. Reference: Lemos, Coogan et al 2023

The TARP diagnostic is a global diagnostic which can be used to check a trained posterior against a set of true values of theta.

Parameters:

  • thetas (Tensor) – ground-truth parameters for tarp, simulated from the prior.

  • xs (Tensor) – observed data for tarp, simulated from thetas.

  • posterior (NeuralPosterior) – a posterior obtained from sbi.

  • num_posterior_samples (int) – number of approximate posterior samples used for ranking.

  • num_workers (int) – number of CPU cores to use for running inference in parallel.

  • show_progress_bar (bool) – whether to display a progress over sbc runs.

  • distance (Callable) – the distance metric to use when computing the distance. Should be a callable function that accepts two tensors and computes the distance between them, e.g. given two tensors of shape (batch, 3) and (batch,3), this function should return (batch,1) distance values. Possible values: sbi.utils.metrics.l1 or sbi.utils.metrics.l2. l2 is the default.

  • num_bins (int | None) – number of bins to use for the credibility values. If None, then num_tarp_samples // 10 bins are used, which targets at least 10 samples per bin (requires num_tarp_samples >= 100).

  • z_score_theta (bool) – whether to normalize parameters before coverage test.

  • use_batched_sampling (bool) – whether to use batched sampling for posterior samples.

  • references (Tensor | None)

Returns:

Expected coverage probability (ecp), see equation 4 of the paper alpha: credibility values, see equation 2 of the paper

Return type:

ecp

Navigation