LC2ST

class LC2ST(thetas, xs, posterior_samples, seed=1, num_folds=1, num_ensemble=1, classifier=<class 'sklearn.neural_network._multilayer_perceptron.MLPClassifier'>, z_score=False, classifier_kwargs=None, num_trials_null=100, permutation=True, device='cpu')

Bases: object

L-C2ST: Local Classifier Two-Sample Test.

Implementation based on the official code from [1] and the exisiting C2ST metric [2], using scikit-learn classifiers.

L-C2ST tests the local consistency of a posterior estimator \(q\) with respect to the true posterior \(p\), at a fixed observation \(x_o\), i.e., whether the following null hypothesis holds:

\(H_0(x_o) := q(\theta \mid x_o) = p(\theta \mid x_o)\).

L-C2ST proceeds as follows:

1. It first trains a classifier to distinguish between samples from two joint distributions \([\theta_p, x_p]\) and \([\theta_q, x_q]\), and evaluates the L-C2ST statistic at a given observation \(x_o\).

2. The L-C2ST statistic is the mean squared error between the predicted probabilities of being in p (class 0) and a Dirac at 0.5, which corresponds to the chance level of the classifier, unable to distinguish between p and q.

• If num_ensemble>1, the average prediction over all classifiers is used.

• If num_folds>1 the average statistic over all cv-folds is used.

To evaluate the test, steps 1 and 2 are performed over multiple trials under the null hypothesis (H0). If the null distribution is not known, it is estimated using the permutation method, i.e. by training the classifier on the permuted data. The statistics obtained under (H0) is then compared to the one obtained on observed data to compute the p-value, used to decide whether to reject (H0) or not.

Parameters:

• thetas (Tensor)

• xs (Tensor)

• posterior_samples (Tensor)

• seed (int)

• num_folds (int)

• num_ensemble (int)

• classifier (str | Type[BaseEstimator])

• z_score (bool)

• classifier_kwargs (Dict[str, Any] | None)

• num_trials_null (int)

• permutation (bool)

• device (str)

get_scores(theta_o, x_o, trained_clfs, return_probs=False)

Computes the L-C2ST scores given the trained classifiers.

Mean squared error (MSE) between 0.5 and the predicted probabilities of being in class 0 over the dataset (theta_o, x_o).

Parameters:

• theta_o (Tensor) – Samples from the posterior conditioned on the observation x_o, of shape (sample_size, dim).

• x_o (Tensor) – The observation, of shape (,dim_x).

• trained_clfs (List[Any]) – List of trained classifiers, of length num_folds.

• return_probs (bool) – Whether to return the predicted probabilities of being in P, defaults to False.

Return type:

ndarray | Tuple[ndarray, ndarray]

Returns: one of

scores: L-C2ST scores at x_o, of shape (num_folds,). (probs, scores): Predicted probabilities and L-C2ST scores at x_o,

each of shape (num_folds,).

train_on_observed_data(seed=None, verbosity=1)

Trains the classifier on the observed data.

Saves the trained classifier(s) as a list of length num_folds.

Parameters:

• seed (int | None) – random state of the classifier, defaults to None.

• verbosity (int) – Verbosity level, defaults to 1.

Return type:

None | List[Any]

get_statistic_on_observed_data(theta_o, x_o)

Computes the L-C2ST statistics for the observed data.

Mean over all cv-scores.

Parameters:

• theta_o (Tensor) – Samples from the posterior conditioned on the observation x_o, of shape (sample_size, dim).

• x_o (Tensor) – The observation, of shape (, dim_x)

Returns:

L-C2ST statistic at x_o.

Return type:

float

p_value(theta_o, x_o)

Computes the p-value for L-C2ST.

The p-value is the proportion of times the L-C2ST statistic under the null hypothesis is greater than the L-C2ST statistic at the observation x_o. It is computed by taking the empirical mean over statistics computed on several trials under the null hypothesis: \(1/H \sum_{h=1}^{H} I(T_h < T_o)\).

Parameters:

• theta_o (Tensor) – Samples from the posterior conditioned on the observation x_o, of dhape (sample_size, dim).

• x_o (Tensor) – The observation, of shape (, dim_x).

Returns:

p-value for L-C2ST at x_o.

Return type:

float

reject_test(theta_o, x_o, alpha=0.05)

Computes the test result for L-C2ST at a given significance level.

Parameters:

• theta_o (Tensor) – Samples from the posterior conditioned on the observation x_o, of shape (sample_size, dim).

• x_o (Tensor) – The observation, of shape (, dim_x).

• alpha (float) – Significance level, defaults to 0.05.

Returns:

True if rejected, False otherwise.

Return type:

The L-C2ST result

train_under_null_hypothesis(verbosity=1)

Computes the L-C2ST scores under the null hypothesis (H0). Saves the trained classifiers for each null trial.

Parameters:

verbosity (int) – Verbosity level, defaults to 1.

Return type:

None

get_statistics_under_null_hypothesis(theta_o, x_o, return_probs=False, verbosity=0)

Computes the L-C2ST scores under the null hypothesis.

Parameters:

• theta_o (Tensor) – Samples from the posterior conditioned on the observation x_o, of shape (sample_size, dim).

• x_o (Tensor) – The observation, of shape (, dim_x).

• return_probs (bool) – Whether to return the predicted probabilities of being in P, defaults to False.

• verbosity (int) – Verbosity level, defaults to 1.

Return type:

ndarray | Tuple[ndarray, ndarray]

Returns: one of

scores: L-C2ST scores under (H0). (probs, scores): Predicted probabilities and L-C2ST scores under (H0).