# conformal.classification¶

Classification module contains methods for conformal classification.

Conformal classifiers predict a set of classes (not always a single class) under a given significance level (error rate). Every classifier works in combination with a nonconformity measure and on average predicts the correct class with the given error rate. Lower error rates result in smaller sets of predicted classes.

Structure:

class conformal.classification.PredictionClass(p, eps)[source]

Bases: object

Conformal classification prediction object, which is produced by the ConformalClassifier.predict() method.

p

List of pairs (p-value, class)

Type: List
eps

Default significance level (error rate).

Type: float

Examples

>>> train, test = next(LOOSampler(Table('iris')))
>>> tcp = TransductiveClassifier(InverseProbability(NaiveBayesLearner()), train)

>>> prediction = tcp.predict(test[0], 0.1)
>>> print(prediction.confidence(), prediction.credibility())

>>> prediction = tcp.predict(test[0])
>>> print(prediction.classes(0.1), prediction.classes(0.9))

__init__(p, eps)[source]

Initialize the prediction.

Parameters: p (List) – List of pairs (p-value, class) eps (float) – Default significance level (error rate).
classes(eps=None)[source]

Compute the set of classes under the default or given eps value.

Parameters: eps (float) – Significance level (error rate). List of predicted classes.
verdict(ref, eps=None)[source]

Conformal classification prediction is correct when the actual class appears among the predicted classes.

Parameters: ref – Reference/actual class eps (float) – Significance level (error rate). True if the prediction with default or specified eps is correct.
confidence()[source]

Confidence is an efficiency measure of a single prediction.

Computes minimum $$\mathit{eps}$$ that would still result in a prediction of a single label. $$\mathit{eps} = \text{second\_largest}(p_i)$$

Returns: Confidence $$1-\mathit{eps}$$. float
credibility()[source]

Credibility is an efficiency measure of a single prediction. Small credibility indicates an unusual example.

Computes minimum $$\mathit{eps}$$ that would result in an empty prediction set. $$\mathit{eps} = \text{max}(p_i)$$

Returns: Credibility $$\mathit{eps}$$. float
class conformal.classification.ConformalClassifier(nc_measure, mondrian=False)[source]

Bases: orangecontrib.conformal.base.ConformalPredictor

Base class for conformal classifiers.

__init__(nc_measure, mondrian=False)[source]

Verify that the nonconformity measure can be used for classification.

p_values(example)[source]

Extending classes should implement this method to return a list of pairs (p-value, class) for a given example.

Conformal classifier assigns an assumed class value to the given example and computes its nonconformity. P-value is the ratio of more nonconformal (stranger) instances that the given example.

predict(example, eps=None)[source]

Compute a classification prediction object from p-values for a given example and significance level.

Parameters: example (Instance) – Orange row instance. eps (float) – Default significance level (error rate). Classification prediction object. PredictionClass
__call__(example, eps)[source]

Compute predicted classes for a given example and significance level.

Parameters: example (Instance) – Orange row instance. eps (float) – Significance level (error rate). List of predicted classes.
class conformal.classification.TransductiveClassifier(nc_measure, train=None, mondrian=False)[source]

Transductive classification.

Examples

>>> train, test = next(LOOSampler(Table('iris')))
>>> tcp = TransductiveClassifier(ProbabilityMargin(NaiveBayesLearner()), train)
>>> print(tcp(test[0], 0.1))

__init__(nc_measure, train=None, mondrian=False)[source]

Initialize transductive classifier with a nonconformity measure and a training set.

Fit the conformal classifier to the training set if present.

Parameters: nc_measure (ClassNC) – Classification nonconformity measure. train (Optional[Table]) – Table of examples used as a training set. mondrian (bool) – Use a mondrian setting for computing p-values.
fit(train)[source]

Fit the conformal classifier to the training set and store the domain.

Parameters: train (Optional[Table]) – Table of examples used as a training set.
p_values(example)[source]

Compute p-values for every possible class.

Transductive classifier appends the given example with an assumed class value to the training set and compares its nonconformity against all other instances.

Parameters: example (Instance) – Orange row instance. List of pairs (p-value, class)
class conformal.classification.InductiveClassifier(nc_measure, train=None, calibrate=None, mondrian=False)[source]

Inductive classification.

alpha

Nonconformity scores of the calibration instances. Computed by the fit() method.

Examples

>>> train, test = next(LOOSampler(Table('iris')))
>>> train, calibrate = next(RandomSampler(train, 2, 1))
>>> icp = InductiveClassifier(InverseProbability(LogisticRegressionLearner()), train, calibrate)
>>> print(icp(test[0], 0.1))

__init__(nc_measure, train=None, calibrate=None, mondrian=False)[source]

Initialize inductive classifier with a nonconformity measure, training set and calibration set. If present, fit the conformal classifier to the training set and compute the nonconformity scores of calibration set.

Parameters: nc_measure (ClassNC) – Classification nonconformity measure. train (Optional[Table]) – Table of examples used as a training set. calibrate (Optional[Table]) – Table of examples used as a calibration set. mondrian (bool) – Use a mondrian setting for computing p-values.
fit(train, calibrate)[source]

Fit the conformal classifier to the training set, compute and store nonconformity scores (alpha) on the calibration set and store the domain.

Parameters: train (Optional[Table]) – Table of examples used as a training set. calibrate (Optional[Table]) – Table of examples used as a calibration set.
p_values(example)[source]

Compute p-values for every possible class.

Inductive classifier assigns an assumed class value to the given example and compares its nonconformity against all other instances in the calibration set.

Parameters: example (Instance) – Orange row instance. List of pairs (p-value, class)
class conformal.classification.CrossClassifier(nc_measure, k, train=None, mondrian=False)[source]

Cross classification.

Examples

>>> train, test = next(LOOSampler(Table('iris')))
>>> ccp = CrossClassifier(InverseProbability(LogisticRegressionLearner()), 3, train)
>>> print(ccp(test[0], 0.1))

__init__(nc_measure, k, train=None, mondrian=False)[source]

Initialize cross classifier with a nonconformity measure, number of folds and training set. If present, fit the conformal classifier to the training set.

Parameters: nc_measure (ClassNC) – Classification nonconformity measure. k (int) – Number of folds. train (Optional[Table]) – Table of examples used as a training set. mondrian (bool) – Use a mondrian setting for computing p-values.
fit(train)[source]

Fit the cross classifier to the training set. Split the training set into k folds for use as training and calibration set with an inductive classifier. Concatenate the computed nonconformity scores and store them (InductiveClassifier.alpha).

Parameters: train (Table) – Table of examples used as a training set.
class conformal.classification.LOOClassifier(nc_measure, train=None, mondrian=False)[source]

Leave-one-out classifier is a cross conformal classifier with the number of folds equal to the size of the training set.

Examples

>>> train, test = next(LOOSampler(Table('iris')))
>>> loocp = LOOClassifier(InverseProbability(LogisticRegressionLearner()), train)
>>> print(loocp(test[0], 0.1))

__init__(nc_measure, train=None, mondrian=False)[source]

Initialize cross classifier with a nonconformity measure, number of folds and training set. If present, fit the conformal classifier to the training set.

Parameters: nc_measure (ClassNC) – Classification nonconformity measure. k (int) – Number of folds. train (Optional[Table]) – Table of examples used as a training set. mondrian (bool) – Use a mondrian setting for computing p-values.
fit(train)[source]

Fit the cross classifier to the training set. Split the training set into k folds for use as training and calibration set with an inductive classifier. Concatenate the computed nonconformity scores and store them (InductiveClassifier.alpha).

Parameters: train (Table) – Table of examples used as a training set.