AUC
classkeras.metrics.AUC(
num_thresholds=200,
curve="ROC",
summation_method="interpolation",
name=None,
dtype=None,
thresholds=None,
multi_label=False,
num_labels=None,
label_weights=None,
from_logits=False,
)
Approximates the AUC (Area under the curve) of the ROC or PR curves.
The AUC (Area under the curve) of the ROC (Receiver operating characteristic; default) or PR (Precision Recall) curves are quality measures of binary classifiers. Unlike the accuracy, and like cross-entropy losses, ROC-AUC and PR-AUC evaluate all the operational points of a model.
This class approximates AUCs using a Riemann sum. During the metric accumulation phrase, predictions are accumulated within predefined buckets by value. The AUC is then computed by interpolating per-bucket averages. These buckets define the evaluated operational points.
This metric creates four local variables, true_positives
,
true_negatives
, false_positives
and false_negatives
that are used to
compute the AUC. To discretize the AUC curve, a linearly spaced set of
thresholds is used to compute pairs of recall and precision values. The area
under the ROC-curve is therefore computed using the height of the recall
values by the false positive rate, while the area under the PR-curve is the
computed using the height of the precision values by the recall.
This value is ultimately returned as auc
, an idempotent operation that
computes the area under a discretized curve of precision versus recall
values (computed using the aforementioned variables). The num_thresholds
variable controls the degree of discretization with larger numbers of
thresholds more closely approximating the true AUC. The quality of the
approximation may vary dramatically depending on num_thresholds
. The
thresholds
parameter can be used to manually specify thresholds which
split the predictions more evenly.
For a best approximation of the real AUC, predictions
should be
distributed approximately uniformly in the range [0, 1]
(if
from_logits=False
). The quality of the AUC approximation may be poor if
this is not the case. Setting summation_method
to 'minoring' or 'majoring'
can help quantify the error in the approximation by providing lower or upper
bound estimate of the AUC.
If sample_weight
is None
, weights default to 1.
Use sample_weight
of 0 to mask values.
Arguments
200
.'ROC'
(default) or 'PR'
for the Precision-Recall-curve.ROC
. For PR-AUC, interpolates (true/false) positives but not
the ratio that is precision (see Davis & Goadrich 2006 for
details); 'minoring' applies left summation for increasing
intervals and right summation for decreasing intervals; 'majoring'
does the opposite.num_thresholds
parameter is ignored. Values should be in [0, 1]
. Endpoint
thresholds equal to {-epsilon
, 1+epsilon
} for a small positive
epsilon value will be automatically included with these to correctly
handle predictions equal to exactly 0 or 1.False
) if the data
should be flattened into a single label before AUC computation. In
the latter case, when multilabel data is passed to AUC, each
label-prediction pair is treated as an individual data point. Should
be set to False
for multi-class data.multi_label
is
True. If num_labels
is not specified, then state variables get
created on the first call to update_state
.multi_label
is
True, the weights are applied to the individual label AUCs when they
are averaged to produce the multi-label AUC. When it's False, they
are used to weight the individual label predictions in computing the
confusion matrix on the flattened data. Note that this is unlike
class_weights
in that class_weights
weights the example
depending on the value of its label, whereas label_weights
depends
only on the index of that label before flattening; therefore
label_weights
should not be used for multi-class data.y_pred
in
update_state
) are probabilities or sigmoid logits. As a rule of thumb,
when using a keras loss, the from_logits
constructor argument of the
loss should match the AUC from_logits
constructor argument.Example
>>> m = keras.metrics.AUC(num_thresholds=3)
>>> m.update_state([0, 0, 1, 1], [0, 0.5, 0.3, 0.9])
>>> # threshold values are [0 - 1e-7, 0.5, 1 + 1e-7]
>>> # tp = [2, 1, 0], fp = [2, 0, 0], fn = [0, 1, 2], tn = [0, 2, 2]
>>> # tp_rate = recall = [1, 0.5, 0], fp_rate = [1, 0, 0]
>>> # auc = ((((1 + 0.5) / 2) * (1 - 0)) + (((0.5 + 0) / 2) * (0 - 0)))
>>> # = 0.75
>>> m.result()
0.75
>>> m.reset_state()
>>> m.update_state([0, 0, 1, 1], [0, 0.5, 0.3, 0.9],
... sample_weight=[1, 0, 0, 1])
>>> m.result()
1.0
Usage with compile()
API:
# Reports the AUC of a model outputting a probability.
model.compile(optimizer='sgd',
loss=keras.losses.BinaryCrossentropy(),
metrics=[keras.metrics.AUC()])
# Reports the AUC of a model outputting a logit.
model.compile(optimizer='sgd',
loss=keras.losses.BinaryCrossentropy(from_logits=True),
metrics=[keras.metrics.AUC(from_logits=True)])
Precision
classkeras.metrics.Precision(
thresholds=None, top_k=None, class_id=None, name=None, dtype=None
)
Computes the precision of the predictions with respect to the labels.
The metric creates two local variables, true_positives
and
false_positives
that are used to compute the precision. This value is
ultimately returned as precision
, an idempotent operation that simply
divides true_positives
by the sum of true_positives
and
false_positives
.
If sample_weight
is None
, weights default to 1.
Use sample_weight
of 0 to mask values.
If top_k
is set, we'll calculate precision as how often on average a class
among the top-k classes with the highest predicted values of a batch entry
is correct and can be found in the label for that entry.
If class_id
is specified, we calculate precision by considering only the
entries in the batch for which class_id
is above the threshold and/or in
the top-k highest predictions, and computing the fraction of them for which
class_id
is indeed a correct label.
Arguments
[0, 1]
. A threshold is compared with
prediction values to determine the truth value of predictions (i.e.,
above the threshold is True
, below is False
). If used with a
loss function that sets from_logits=True
(i.e. no sigmoid applied
to predictions), thresholds
should be set to 0. One metric value
is generated for each threshold value. If neither thresholds
nor
top_k
are set, the default is to calculate precision with
thresholds=0.5
.[0, num_classes)
, where
num_classes
is the last dimension of predictions.Example
>>> m = keras.metrics.Precision()
>>> m.update_state([0, 1, 1, 1], [1, 0, 1, 1])
>>> m.result()
0.6666667
>>> m.reset_state()
>>> m.update_state([0, 1, 1, 1], [1, 0, 1, 1], sample_weight=[0, 0, 1, 0])
>>> m.result()
1.0
>>> # With top_k=2, it will calculate precision over y_true[:2]
>>> # and y_pred[:2]
>>> m = keras.metrics.Precision(top_k=2)
>>> m.update_state([0, 0, 1, 1], [1, 1, 1, 1])
>>> m.result()
0.0
>>> # With top_k=4, it will calculate precision over y_true[:4]
>>> # and y_pred[:4]
>>> m = keras.metrics.Precision(top_k=4)
>>> m.update_state([0, 0, 1, 1], [1, 1, 1, 1])
>>> m.result()
0.5
Usage with compile()
API:
model.compile(optimizer='sgd',
loss='binary_crossentropy',
metrics=[keras.metrics.Precision()])
Usage with a loss with from_logits=True
:
model.compile(optimizer='adam',
loss=keras.losses.BinaryCrossentropy(from_logits=True),
metrics=[keras.metrics.Precision(thresholds=0)])
Recall
classkeras.metrics.Recall(
thresholds=None, top_k=None, class_id=None, name=None, dtype=None
)
Computes the recall of the predictions with respect to the labels.
This metric creates two local variables, true_positives
and
false_negatives
, that are used to compute the recall. This value is
ultimately returned as recall
, an idempotent operation that simply divides
true_positives
by the sum of true_positives
and false_negatives
.
If sample_weight
is None
, weights default to 1.
Use sample_weight
of 0 to mask values.
If top_k
is set, recall will be computed as how often on average a class
among the labels of a batch entry is in the top-k predictions.
If class_id
is specified, we calculate recall by considering only the
entries in the batch for which class_id
is in the label, and computing the
fraction of them for which class_id
is above the threshold and/or in the
top-k predictions.
Arguments
[0, 1]
. A threshold is compared with
prediction values to determine the truth value of predictions (i.e.,
above the threshold is True
, below is False
). If used with a
loss function that sets from_logits=True
(i.e. no sigmoid
applied to predictions), thresholds
should be set to 0.
One metric value is generated for each threshold value.
If neither thresholds
nor top_k
are set,
the default is to calculate recall with thresholds=0.5
.[0, num_classes)
, where
num_classes
is the last dimension of predictions.Example
>>> m = keras.metrics.Recall()
>>> m.update_state([0, 1, 1, 1], [1, 0, 1, 1])
>>> m.result()
0.6666667
>>> m.reset_state()
>>> m.update_state([0, 1, 1, 1], [1, 0, 1, 1], sample_weight=[0, 0, 1, 0])
>>> m.result()
1.0
Usage with compile()
API:
model.compile(optimizer='sgd',
loss='binary_crossentropy',
metrics=[keras.metrics.Recall()])
Usage with a loss with from_logits=True
:
model.compile(optimizer='adam',
loss=keras.losses.BinaryCrossentropy(from_logits=True),
metrics=[keras.metrics.Recall(thresholds=0)])
TruePositives
classkeras.metrics.TruePositives(thresholds=None, name=None, dtype=None)
Calculates the number of true positives.
If sample_weight
is given, calculates the sum of the weights of
true positives. This metric creates one local variable, true_positives
that is used to keep track of the number of true positives.
If sample_weight
is None
, weights default to 1.
Use sample_weight
of 0 to mask values.
Arguments
0.5
. A float value, or a Python
list/tuple of float threshold values in [0, 1]
. A threshold is
compared with prediction values to determine the truth value of
predictions (i.e., above the threshold is True
, below is False
).
If used with a loss function that sets from_logits=True
(i.e. no
sigmoid applied to predictions), thresholds
should be set to 0.
One metric value is generated for each threshold value.Example
>>> m = keras.metrics.TruePositives()
>>> m.update_state([0, 1, 1, 1], [1, 0, 1, 1])
>>> m.result()
2.0
>>> m.reset_state()
>>> m.update_state([0, 1, 1, 1], [1, 0, 1, 1], sample_weight=[0, 0, 1, 0])
>>> m.result()
1.0
TrueNegatives
classkeras.metrics.TrueNegatives(thresholds=None, name=None, dtype=None)
Calculates the number of true negatives.
If sample_weight
is given, calculates the sum of the weights of
true negatives. This metric creates one local variable, accumulator
that is used to keep track of the number of true negatives.
If sample_weight
is None
, weights default to 1.
Use sample_weight
of 0 to mask values.
Arguments
0.5
. A float value, or a Python
list/tuple of float threshold values in [0, 1]
. A threshold is
compared with prediction values to determine the truth value of
predictions (i.e., above the threshold is True
, below is False
).
If used with a loss function that sets from_logits=True
(i.e. no
sigmoid applied to predictions), thresholds
should be set to 0.
One metric value is generated for each threshold value.Example
>>> m = keras.metrics.TrueNegatives()
>>> m.update_state([0, 1, 0, 0], [1, 1, 0, 0])
>>> m.result()
2.0
>>> m.reset_state()
>>> m.update_state([0, 1, 0, 0], [1, 1, 0, 0], sample_weight=[0, 0, 1, 0])
>>> m.result()
1.0
FalsePositives
classkeras.metrics.FalsePositives(thresholds=None, name=None, dtype=None)
Calculates the number of false positives.
If sample_weight
is given, calculates the sum of the weights of
false positives. This metric creates one local variable, accumulator
that is used to keep track of the number of false positives.
If sample_weight
is None
, weights default to 1.
Use sample_weight
of 0 to mask values.
Arguments
0.5
. A float value, or a Python
list/tuple of float threshold values in [0, 1]
. A threshold is
compared with prediction values to determine the truth value of
predictions (i.e., above the threshold is True
, below is False
).
If used with a loss function that sets from_logits=True
(i.e. no
sigmoid applied to predictions), thresholds
should be set to 0.
One metric value is generated for each threshold value.Examples
>>> m = keras.metrics.FalsePositives()
>>> m.update_state([0, 1, 0, 0], [0, 0, 1, 1])
>>> m.result()
2.0
>>> m.reset_state()
>>> m.update_state([0, 1, 0, 0], [0, 0, 1, 1], sample_weight=[0, 0, 1, 0])
>>> m.result()
1.0
FalseNegatives
classkeras.metrics.FalseNegatives(thresholds=None, name=None, dtype=None)
Calculates the number of false negatives.
If sample_weight
is given, calculates the sum of the weights of
false negatives. This metric creates one local variable, accumulator
that is used to keep track of the number of false negatives.
If sample_weight
is None
, weights default to 1.
Use sample_weight
of 0 to mask values.
Arguments
0.5
. A float value, or a Python
list/tuple of float threshold values in [0, 1]
. A threshold is
compared with prediction values to determine the truth value of
predictions (i.e., above the threshold is True
, below is False
).
If used with a loss function that sets from_logits=True
(i.e. no
sigmoid applied to predictions), thresholds
should be set to 0.
One metric value is generated for each threshold value.Example
>>> m = keras.metrics.FalseNegatives()
>>> m.update_state([0, 1, 1, 1], [0, 1, 0, 0])
>>> m.result()
2.0
>>> m.reset_state()
>>> m.update_state([0, 1, 1, 1], [0, 1, 0, 0], sample_weight=[0, 0, 1, 0])
>>> m.result()
1.0
PrecisionAtRecall
classkeras.metrics.PrecisionAtRecall(
recall, num_thresholds=200, class_id=None, name=None, dtype=None
)
Computes best precision where recall is >= specified value.
This metric creates four local variables, true_positives
,
true_negatives
, false_positives
and false_negatives
that are used to
compute the precision at the given recall. The threshold for the given
recall value is computed and used to evaluate the corresponding precision.
If sample_weight
is None
, weights default to 1.
Use sample_weight
of 0 to mask values.
If class_id
is specified, we calculate precision by considering only the
entries in the batch for which class_id
is above the threshold
predictions, and computing the fraction of them for which class_id
is
indeed a correct label.
Arguments
[0, 1]
.[0, num_classes)
, where
num_classes
is the last dimension of predictions.Example
>>> m = keras.metrics.PrecisionAtRecall(0.5)
>>> m.update_state([0, 0, 0, 1, 1], [0, 0.3, 0.8, 0.3, 0.8])
>>> m.result()
0.5
>>> m.reset_state()
>>> m.update_state([0, 0, 0, 1, 1], [0, 0.3, 0.8, 0.3, 0.8],
... sample_weight=[2, 2, 2, 1, 1])
>>> m.result()
0.33333333
Usage with compile()
API:
model.compile(
optimizer='sgd',
loss='binary_crossentropy',
metrics=[keras.metrics.PrecisionAtRecall(recall=0.8)])
RecallAtPrecision
classkeras.metrics.RecallAtPrecision(
precision, num_thresholds=200, class_id=None, name=None, dtype=None
)
Computes best recall where precision is >= specified value.
For a given score-label-distribution the required precision might not be achievable, in this case 0.0 is returned as recall.
This metric creates four local variables, true_positives
,
true_negatives
, false_positives
and false_negatives
that are used to
compute the recall at the given precision. The threshold for the given
precision value is computed and used to evaluate the corresponding recall.
If sample_weight
is None
, weights default to 1.
Use sample_weight
of 0 to mask values.
If class_id
is specified, we calculate precision by considering only the
entries in the batch for which class_id
is above the threshold
predictions, and computing the fraction of them for which class_id
is
indeed a correct label.
Arguments
[0, 1]
.[0, num_classes)
, where
num_classes
is the last dimension of predictions.Example
>>> m = keras.metrics.RecallAtPrecision(0.8)
>>> m.update_state([0, 0, 1, 1], [0, 0.5, 0.3, 0.9])
>>> m.result()
0.5
>>> m.reset_state()
>>> m.update_state([0, 0, 1, 1], [0, 0.5, 0.3, 0.9],
... sample_weight=[1, 0, 0, 1])
>>> m.result()
1.0
Usage with compile()
API:
model.compile(
optimizer='sgd',
loss='binary_crossentropy',
metrics=[keras.metrics.RecallAtPrecision(precision=0.8)])
SensitivityAtSpecificity
classkeras.metrics.SensitivityAtSpecificity(
specificity, num_thresholds=200, class_id=None, name=None, dtype=None
)
Computes best sensitivity where specificity is >= specified value.
Sensitivity
measures the proportion of actual positives that are correctly
identified as such (tp / (tp + fn))
.
Specificity
measures the proportion of actual negatives that are correctly
identified as such (tn / (tn + fp))
.
This metric creates four local variables, true_positives
,
true_negatives
, false_positives
and false_negatives
that are used to
compute the sensitivity at the given specificity. The threshold for the
given specificity value is computed and used to evaluate the corresponding
sensitivity.
If sample_weight
is None
, weights default to 1.
Use sample_weight
of 0 to mask values.
If class_id
is specified, we calculate precision by considering only the
entries in the batch for which class_id
is above the threshold
predictions, and computing the fraction of them for which class_id
is
indeed a correct label.
For additional information about specificity and sensitivity, see the following.
Arguments
[0, 1]
.[0, num_classes)
, where
num_classes
is the last dimension of predictions.Example
>>> m = keras.metrics.SensitivityAtSpecificity(0.5)
>>> m.update_state([0, 0, 0, 1, 1], [0, 0.3, 0.8, 0.3, 0.8])
>>> m.result()
0.5
>>> m.reset_state()
>>> m.update_state([0, 0, 0, 1, 1], [0, 0.3, 0.8, 0.3, 0.8],
... sample_weight=[1, 1, 2, 2, 1])
>>> m.result()
0.333333
Usage with compile()
API:
model.compile(
optimizer='sgd',
loss='binary_crossentropy',
metrics=[keras.metrics.SensitivityAtSpecificity()])
SpecificityAtSensitivity
classkeras.metrics.SpecificityAtSensitivity(
sensitivity, num_thresholds=200, class_id=None, name=None, dtype=None
)
Computes best specificity where sensitivity is >= specified value.
Sensitivity
measures the proportion of actual positives that are correctly
identified as such (tp / (tp + fn))
.
Specificity
measures the proportion of actual negatives that are correctly
identified as such (tn / (tn + fp))
.
This metric creates four local variables, true_positives
,
true_negatives
, false_positives
and false_negatives
that are used to
compute the specificity at the given sensitivity. The threshold for the
given sensitivity value is computed and used to evaluate the corresponding
specificity.
If sample_weight
is None
, weights default to 1.
Use sample_weight
of 0 to mask values.
If class_id
is specified, we calculate precision by considering only the
entries in the batch for which class_id
is above the threshold
predictions, and computing the fraction of them for which class_id
is
indeed a correct label.
For additional information about specificity and sensitivity, see the following.
Arguments
[0, 1]
.[0, num_classes)
, where
num_classes
is the last dimension of predictions.Example
>>> m = keras.metrics.SpecificityAtSensitivity(0.5)
>>> m.update_state([0, 0, 0, 1, 1], [0, 0.3, 0.8, 0.3, 0.8])
>>> m.result()
0.66666667
>>> m.reset_state()
>>> m.update_state([0, 0, 0, 1, 1], [0, 0.3, 0.8, 0.3, 0.8],
... sample_weight=[1, 1, 2, 2, 2])
>>> m.result()
0.5
Usage with compile()
API:
model.compile(
optimizer='sgd',
loss='binary_crossentropy',
metrics=[keras.metrics.SpecificityAtSensitivity()])
F1Score
classkeras.metrics.F1Score(average=None, threshold=None, name="f1_score", dtype=None)
Computes F-1 Score.
Formula:
f1_score = 2 * (precision * recall) / (precision + recall)
This is the harmonic mean of precision and recall.
Its output range is [0, 1]
. It works for both multi-class
and multi-label classification.
Arguments
None
, "micro"
, "macro"
and "weighted"
. Defaults to None
.
If None
, no averaging is performed and result()
will return
the score for each class.
If "micro"
, compute metrics globally by counting the total
true positives, false negatives and false positives.
If "macro"
, compute metrics for each label,
and return their unweighted mean.
This does not take label imbalance into account.
If "weighted"
, compute metrics for each label,
and return their average weighted by support
(the number of true instances for each label).
This alters "macro"
to account for label imbalance.
It can result in an score that is not between precision and recall.y_pred
greater than threshold
are
converted to be 1, and the rest 0. If threshold
is
None
, the argmax of y_pred
is converted to 1, and the rest to 0.Returns
Example
>>> metric = keras.metrics.F1Score(threshold=0.5)
>>> y_true = np.array([[1, 1, 1],
... [1, 0, 0],
... [1, 1, 0]], np.int32)
>>> y_pred = np.array([[0.2, 0.6, 0.7],
... [0.2, 0.6, 0.6],
... [0.6, 0.8, 0.0]], np.float32)
>>> metric.update_state(y_true, y_pred)
>>> result = metric.result()
array([0.5 , 0.8 , 0.6666667], dtype=float32)
FBetaScore
classkeras.metrics.FBetaScore(
average=None, beta=1.0, threshold=None, name="fbeta_score", dtype=None
)
Computes F-Beta score.
Formula:
b2 = beta ** 2
f_beta_score = (1 + b2) * (precision * recall) / (precision * b2 + recall)
This is the weighted harmonic mean of precision and recall.
Its output range is [0, 1]
. It works for both multi-class
and multi-label classification.
Arguments
None
, "micro"
, "macro"
and
"weighted"
. Defaults to None
.
If None
, no averaging is performed and result()
will return
the score for each class.
If "micro"
, compute metrics globally by counting the total
true positives, false negatives and false positives.
If "macro"
, compute metrics for each label,
and return their unweighted mean.
This does not take label imbalance into account.
If "weighted"
, compute metrics for each label,
and return their average weighted by support
(the number of true instances for each label).
This alters "macro"
to account for label imbalance.
It can result in an score that is not between precision and recall.1
.y_pred
greater than threshold
are
converted to be 1, and the rest 0. If threshold
is
None
, the argmax of y_pred
is converted to 1, and the rest to 0.Returns
Example
>>> metric = keras.metrics.FBetaScore(beta=2.0, threshold=0.5)
>>> y_true = np.array([[1, 1, 1],
... [1, 0, 0],
... [1, 1, 0]], np.int32)
>>> y_pred = np.array([[0.2, 0.6, 0.7],
... [0.2, 0.6, 0.6],
... [0.6, 0.8, 0.0]], np.float32)
>>> metric.update_state(y_true, y_pred)
>>> result = metric.result()
>>> result
[0.3846154 , 0.90909094, 0.8333334 ]