LC2ST

LC2ST#

class LC2ST(prior_samples=None, xs=None, posterior_samples=None, seed=1, num_folds=1, num_ensemble=1, classifier=<class 'sklearn.neural_network._multilayer_perceptron.MLPClassifier'>, z_score=False, classifier_kwargs=None, num_trials_null=100, permutation=True, device='cpu', *, thetas=None)[source]#

Bases: object

L-C2ST: Local Classifier Two-Sample Test.

Implementation based on the official code from [1] and the existing C2ST metric [2], using scikit-learn classifiers.

L-C2ST tests the local consistency of a posterior estimator \(q\) with respect to the true posterior \(p\), at a fixed observation \(x_o\), i.e., whether the following null hypothesis holds:

\(H_0(x_o) := q(\theta \mid x_o) = p(\theta \mid x_o)\).

L-C2ST proceeds as follows:

  1. It first trains a classifier to distinguish between samples from two joint distributions \([\theta_p, x_p]\) and \([\theta_q, x_q]\), and evaluates the L-C2ST statistic at a given observation \(x_o\).

  2. The L-C2ST statistic is the mean squared error between the predicted probabilities of being in p (class 0) and a Dirac at 0.5, which corresponds to the chance level of the classifier, unable to distinguish between p and q.

  • If num_ensemble>1, the average prediction over all classifiers is used.

  • If num_folds>1 the average statistic over all cv-folds is used.

To evaluate the test, steps 1 and 2 are performed over multiple trials under the null hypothesis (H0). If the null distribution is not known, it is estimated using the permutation method, i.e. by training the classifier on the permuted data. The statistics obtained under (H0) is then compared to the one obtained on observed data to compute the p-value, used to decide whether to reject (H0) or not.

Example

>>> lc2st = LC2ST(prior_samples, xs, posterior_samples, num_folds=5)
>>> lc2st.train_on_observed_data().train_under_null_hypothesis()
>>> p_value = lc2st.p_value(theta_o=theta_obs, x_o=x_obs)
>>> rejected = lc2st.reject_test(theta_o=theta_obs, x_o=x_obs, alpha=0.05)

Note

Training methods can be called in either order:

  • train_on_observed_data().train_under_null_hypothesis() (recommended)

  • train_under_null_hypothesis().train_on_observed_data() (also valid)

Both orders reach the READY state required for p_value() and reject_test().

References

[1] Linhart et al. (2023), https://arxiv.org/abs/2306.03580 [2] Lopez-Paz & Oquab (2017), https://openreview.net/forum?id=SJkXfE5xx

Parameters:
property state: LC2STState#

Return the current state of the LC2ST object.

get_scores(theta_o, x_o, trained_clfs, return_probs=False)[source]#

Computes the L-C2ST scores given the trained classifiers.

Mean squared error (MSE) between 0.5 and the predicted probabilities of being in class 0 over the dataset (theta_o, x_o).

Parameters:

  • theta_o (Tensor) – Samples from the posterior conditioned on the observation x_o, of shape (sample_size, dim).

  • x_o (Tensor) – The observation, of shape (,dim_x).

  • trained_clfs (List[BaseEstimator]) – List of trained classifiers, of length num_folds.

  • return_probs (bool) – Whether to return the predicted probabilities of being in P, defaults to False. Deprecated: Use the LC2STScores.probabilities attribute instead.

Returns:

LC2STScores object containing scores and optionally probabilities. For backward compatibility, if return_probs=True, returns a tuple (probs, scores) instead.

Return type:

LC2STScores | Tuple[ndarray, ndarray]

train_on_observed_data(seed=None, verbosity=1)[source]#

Trains the classifier on the observed data.

Saves the trained classifier(s) as a list of length num_folds.

Parameters:

  • seed (int | None) – Random state of the classifier, defaults to None.

  • verbosity (int) – Verbosity level, defaults to 1.

Returns:

self, for method chaining.

Return type:

LC2ST

get_statistic_on_observed_data(theta_o, x_o)[source]#

Computes the L-C2ST statistics for the observed data.

Mean over all cv-scores.

Parameters:

  • theta_o (Tensor) – Samples from the posterior conditioned on the observation x_o, of shape (sample_size, dim).

  • x_o (Tensor) – The observation, of shape (, dim_x)

Returns:

L-C2ST statistic at x_o.

Raises:

RuntimeError – If classifiers have not been trained on observed data.

Return type:

float

p_value(theta_o, x_o)[source]#

Computes the p-value for L-C2ST.

The p-value is the proportion of times the L-C2ST statistic under the null hypothesis is greater than the L-C2ST statistic at the observation x_o. It is computed by taking the empirical mean over statistics computed on several trials under the null hypothesis: \(1/H \sum_{h=1}^{H} I(T_h < T_o)\).

Parameters:

  • theta_o (Tensor) – Samples from the posterior conditioned on the observation x_o, of shape (sample_size, dim).

  • x_o (Tensor) – The observation, of shape (, dim_x).

Returns:

p-value for L-C2ST at x_o.

Raises:

RuntimeError – If the LC2ST is not in READY state (both training methods must be called first).

Return type:

float

reject_test(theta_o, x_o, alpha=0.05)[source]#

Computes the test result for L-C2ST at a given significance level.

Parameters:

  • theta_o (Tensor) – Samples from the posterior conditioned on the observation x_o, of shape (sample_size, dim).

  • x_o (Tensor) – The observation, of shape (, dim_x).

  • alpha (float) – Significance level, defaults to 0.05.

Returns:

True if rejected, False otherwise.

Return type:

The L-C2ST result

train_under_null_hypothesis(verbosity=1)[source]#

Computes the L-C2ST scores under the null hypothesis (H0).

Saves the trained classifiers for each null trial.

Parameters:

verbosity (int) – Verbosity level, defaults to 1.

Returns:

self, for method chaining.

Raises:

ValueError – If permutation is False but no null distribution is set.

Return type:

LC2ST

get_statistics_under_null_hypothesis(theta_o, x_o, return_probs=False, verbosity=0)[source]#

Computes the L-C2ST scores under the null hypothesis.

Parameters:

  • theta_o (Tensor) – Samples from the posterior conditioned on the observation x_o, of shape (sample_size, dim).

  • x_o (Tensor) – The observation, of shape (, dim_x).

  • return_probs (bool) – Whether to return the predicted probabilities of being in P, defaults to False.

  • verbosity (int) – Verbosity level, defaults to 0.

Returns:

LC2STScores object containing null statistics and probabilities. For backward compatibility, if return_probs=True, returns a tuple (probs, scores) instead.

Raises:

  • RuntimeError – If classifiers have not been trained under null hypothesis.

  • ValueError – If null distribution required but not set.

Return type:

LC2STScores | Tuple[ndarray, ndarray]

Navigation