conformal.classification

Classification module contains methods for conformal classification.

Conformal classifiers predict a set of classes (not always a single class) under a given significance level (error rate). Every classifier works in combination with a nonconformity measure and on average predicts the correct class with the given error rate. Lower error rates result in smaller sets of predicted classes.

Structure:

• ConformalClassifier

  • Transductive (TransductiveClassifier)
  • Inductive (InductiveClassifier)
  • Cross (CrossClassifier)

class conformal.classification.PredictionClass(p, eps)

Bases: object

Conformal classification prediction object, which is produced by the ConformalClassifier.predict() method.

p

List of pairs (p-value, class)

Type:List

eps

Default significance level (error rate).

Type:float

Examples

>>> train, test = next(LOOSampler(Table('iris')))
>>> tcp = TransductiveClassifier(InverseProbability(NaiveBayesLearner()), train)

>>> prediction = tcp.predict(test[0], 0.1)
>>> print(prediction.confidence(), prediction.credibility())

>>> prediction = tcp.predict(test[0])
>>> print(prediction.classes(0.1), prediction.classes(0.9))

__init__(p, eps)

Initialize the prediction.

Parameters:

• p (List) – List of pairs (p-value, class)
• eps (float) – Default significance level (error rate).

classes(eps=None)

Compute the set of classes under the default or given eps value.

Parameters:eps (float) – Significance level (error rate). Returns:List of predicted classes.

verdict(ref, eps=None)

Conformal classification prediction is correct when the actual class appears among the predicted classes.

Parameters:

• ref – Reference/actual class
• eps (float) – Significance level (error rate).

Returns:

True if the prediction with default or specified eps is correct.

confidence()

Confidence is an efficiency measure of a single prediction.

Computes minimum \(\mathit{eps}\) that would still result in a prediction of a single label. \(\mathit{eps} = \text{second\_largest}(p_i)\)

Returns:Confidence \(1-\mathit{eps}\). Return type:float

credibility()

Credibility is an efficiency measure of a single prediction. Small credibility indicates an unusual example.

Computes minimum \(\mathit{eps}\) that would result in an empty prediction set. \(\mathit{eps} = \text{max}(p_i)\)

Returns:Credibility \(\mathit{eps}\). Return type:float

class conformal.classification.ConformalClassifier(nc_measure, mondrian=False)

Bases: orangecontrib.conformal.base.ConformalPredictor

Base class for conformal classifiers.

__init__(nc_measure, mondrian=False)

Verify that the nonconformity measure can be used for classification.

p_values(example)

Extending classes should implement this method to return a list of pairs (p-value, class) for a given example.

Conformal classifier assigns an assumed class value to the given example and computes its nonconformity. P-value is the ratio of more nonconformal (stranger) instances that the given example.

predict(example, eps=None)

Compute a classification prediction object from p-values for a given example and significance level.

Parameters:

• example (Instance) – Orange row instance.
• eps (float) – Default significance level (error rate).

Returns:

Classification prediction object.

Return type:

PredictionClass

__call__(example, eps)

Compute predicted classes for a given example and significance level.

Parameters:

• example (Instance) – Orange row instance.
• eps (float) – Significance level (error rate).

Returns:

List of predicted classes.

class conformal.classification.TransductiveClassifier(nc_measure, train=None, mondrian=False)

Bases: conformal.classification.ConformalClassifier

Transductive classification.

Examples

>>> train, test = next(LOOSampler(Table('iris')))
>>> tcp = TransductiveClassifier(ProbabilityMargin(NaiveBayesLearner()), train)
>>> print(tcp(test[0], 0.1))

__init__(nc_measure, train=None, mondrian=False)

Initialize transductive classifier with a nonconformity measure and a training set.

Fit the conformal classifier to the training set if present.

Parameters:

• nc_measure (ClassNC) – Classification nonconformity measure.
• train (Optional[Table]) – Table of examples used as a training set.
• mondrian (bool) – Use a mondrian setting for computing p-values.

fit(train)

Fit the conformal classifier to the training set and store the domain.

Parameters:train (Optional[Table]) – Table of examples used as a training set.

p_values(example)

Compute p-values for every possible class.

Transductive classifier appends the given example with an assumed class value to the training set and compares its nonconformity against all other instances.

Parameters:example (Instance) – Orange row instance. Returns:List of pairs (p-value, class)

class conformal.classification.InductiveClassifier(nc_measure, train=None, calibrate=None, mondrian=False)

Bases: conformal.classification.ConformalClassifier

Inductive classification.

alpha

Nonconformity scores of the calibration instances. Computed by the fit() method.

Examples

>>> train, test = next(LOOSampler(Table('iris')))
>>> train, calibrate = next(RandomSampler(train, 2, 1))
>>> icp = InductiveClassifier(InverseProbability(LogisticRegressionLearner()), train, calibrate)
>>> print(icp(test[0], 0.1))

__init__(nc_measure, train=None, calibrate=None, mondrian=False)

Initialize inductive classifier with a nonconformity measure, training set and calibration set. If present, fit the conformal classifier to the training set and compute the nonconformity scores of calibration set.

Parameters:

• nc_measure (ClassNC) – Classification nonconformity measure.
• train (Optional[Table]) – Table of examples used as a training set.
• calibrate (Optional[Table]) – Table of examples used as a calibration set.
• mondrian (bool) – Use a mondrian setting for computing p-values.

fit(train, calibrate)

Fit the conformal classifier to the training set, compute and store nonconformity scores (alpha) on the calibration set and store the domain.

Parameters:

• train (Optional[Table]) – Table of examples used as a training set.
• calibrate (Optional[Table]) – Table of examples used as a calibration set.

p_values(example)

Compute p-values for every possible class.

Inductive classifier assigns an assumed class value to the given example and compares its nonconformity against all other instances in the calibration set.

Parameters:example (Instance) – Orange row instance. Returns:List of pairs (p-value, class)

class conformal.classification.CrossClassifier(nc_measure, k, train=None, mondrian=False)

Bases: conformal.classification.InductiveClassifier

Cross classification.

Examples

>>> train, test = next(LOOSampler(Table('iris')))
>>> ccp = CrossClassifier(InverseProbability(LogisticRegressionLearner()), 3, train)
>>> print(ccp(test[0], 0.1))

__init__(nc_measure, k, train=None, mondrian=False)

Initialize cross classifier with a nonconformity measure, number of folds and training set. If present, fit the conformal classifier to the training set.

Parameters:

• nc_measure (ClassNC) – Classification nonconformity measure.
• k (int) – Number of folds.
• train (Optional[Table]) – Table of examples used as a training set.
• mondrian (bool) – Use a mondrian setting for computing p-values.

fit(train)

Fit the cross classifier to the training set. Split the training set into k folds for use as training and calibration set with an inductive classifier. Concatenate the computed nonconformity scores and store them (InductiveClassifier.alpha).

Parameters:train (Table) – Table of examples used as a training set.

class conformal.classification.LOOClassifier(nc_measure, train=None, mondrian=False)

Bases: conformal.classification.CrossClassifier

Leave-one-out classifier is a cross conformal classifier with the number of folds equal to the size of the training set.

Examples

>>> train, test = next(LOOSampler(Table('iris')))
>>> loocp = LOOClassifier(InverseProbability(LogisticRegressionLearner()), train)
>>> print(loocp(test[0], 0.1))

__init__(nc_measure, train=None, mondrian=False)

Initialize cross classifier with a nonconformity measure, number of folds and training set. If present, fit the conformal classifier to the training set.

Parameters:

• nc_measure (ClassNC) – Classification nonconformity measure.
• k (int) – Number of folds.
• train (Optional[Table]) – Table of examples used as a training set.
• mondrian (bool) – Use a mondrian setting for computing p-values.

fit(train)

Fit the cross classifier to the training set. Split the training set into k folds for use as training and calibration set with an inductive classifier. Concatenate the computed nonconformity scores and store them (InductiveClassifier.alpha).

Parameters:train (Table) – Table of examples used as a training set.