xbe.at

Machine learning / Model evaluation and validation

Introduction to Precision-Recall Curve in Python

6 min read

Introduction to Precision-Recall Curve

The Precision-Recall Curve is a powerful evaluation metric used in machine learning, particularly in classification tasks. It provides a way to assess the performance of a binary classifier by visualizing the trade-off between precision and recall at different classification thresholds.

Code:

# No code for the introduction slide

Understanding Precision and Recall

Precision is the ratio of true positive predictions to the total positive predictions made by the model. Recall, on the other hand, is the ratio of true positive predictions to the total number of actual positive instances.

Code:

from sklearn.metrics import precision_score, recall_score

# Assuming y_true and y_pred are the actual and predicted labels, respectively
precision = precision_score(y_true, y_pred)
recall = recall_score(y_true, y_pred)

Precision-Recall Curve: Concept

The Precision-Recall Curve plots precision on the y-axis and recall on the x-axis for different classification thresholds. By adjusting the threshold, the model can trade precision for recall, or vice versa, depending on the specific requirements of the task.

Code:

from sklearn.metrics import precision_recall_curve
import matplotlib.pyplot as plt

# Assuming y_true and y_score are the actual labels and predicted scores, respectively
precision, recall, thresholds = precision_recall_curve(y_true, y_score)

plt.plot(recall, precision)
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.title('Precision-Recall Curve')
plt.show()

Interpreting the Precision-Recall Curve

A perfect classifier will have a Precision-Recall Curve that hugs the top-right corner, indicating high precision and recall values. The closer the curve is to the top-right corner, the better the performance of the classifier.

Code:

import numpy as np
from sklearn.metrics import precision_recall_curve
import matplotlib.pyplot as plt

# Assuming y_true and y_score are the actual labels and predicted scores, respectively
precision, recall, thresholds = precision_recall_curve(y_true, y_score)

# Plot the Precision-Recall Curve
plt.plot(recall, precision, label='Classifier')

# Plot the random classifier baseline (assume equal class distribution)
random_baseline = np.linspace(0, 1, len(precision))
plt.plot(random_baseline, random_baseline, linestyle='--', label='Random Baseline')

plt.xlabel('Recall')
plt.ylabel('Precision')
plt.title('Precision-Recall Curve')
plt.legend()
plt.show()

Calculating Precision-Recall Curve with scikit-learn

The scikit-learn library in Python provides a convenient function to calculate the Precision-Recall Curve for a given set of true labels and predicted scores.

Code:

from sklearn.metrics import precision_recall_curve
import matplotlib.pyplot as plt

# Assuming y_true and y_score are the actual labels and predicted scores, respectively
precision, recall, thresholds = precision_recall_curve(y_true, y_score)

# Plot the Precision-Recall Curve
plt.plot(recall, precision)
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.title('Precision-Recall Curve')
plt.show()

Adjusting Classification Thresholds

The Precision-Recall Curve allows you to choose an appropriate classification threshold based on the desired trade-off between precision and recall for your specific problem.

Code:

from sklearn.metrics import precision_recall_curve

# Assuming y_true and y_score are the actual labels and predicted scores, respectively
precision, recall, thresholds = precision_recall_curve(y_true, y_score)

# Select a desired threshold value
desired_threshold = 0.5
precision_at_threshold = precision[thresholds >= desired_threshold][-1]
recall_at_threshold = recall[thresholds >= desired_threshold][-1]

print(f"At threshold {desired_threshold}, precision is {precision_at_threshold:.2f} and recall is {recall_at_threshold:.2f}")

Area Under the Precision-Recall Curve (AUPRC)

The Area Under the Precision-Recall Curve (AUPRC) is a single scalar value that summarizes the Precision-Recall Curve. It provides an overall performance metric for the classifier, with higher values indicating better performance.

Code:

from sklearn.metrics import precision_recall_curve, average_precision_score

# Assuming y_true and y_score are the actual labels and predicted scores, respectively
precision, recall, _ = precision_recall_curve(y_true, y_score)
auprc = average_precision_score(y_true, y_score)

print(f"Area Under the Precision-Recall Curve (AUPRC): {auprc:.2f}")

Comparing Classifiers with Precision-Recall Curve

The Precision-Recall Curve can be used to compare the performance of different classifiers on the same dataset. The classifier with the curve closer to the top-right corner is considered better.

Code:

from sklearn.metrics import precision_recall_curve
import matplotlib.pyplot as plt

# Assuming y_true, y_score_clf1, and y_score_clf2 are the actual labels, and predicted scores for two classifiers, respectively
precision_clf1, recall_clf1, _ = precision_recall_curve(y_true, y_score_clf1)
precision_clf2, recall_clf2, _ = precision_recall_curve(y_true, y_score_clf2)

plt.plot(recall_clf1, precision_clf1, label='Classifier 1')
plt.plot(recall_clf2, precision_clf2, label='Classifier 2')
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.title('Precision-Recall Curve Comparison')
plt.legend()
plt.show()

Precision-Recall Curve for Multi-Class Classification

The Precision-Recall Curve can be extended to multi-class classification problems by calculating the curve for each class individually and then averaging the results or using a micro-averaging or macro-averaging approach.

Code:

from sklearn.metrics import precision_recall_curve
import matplotlib.pyplot as plt

# Assuming y_true and y_score are the actual labels and predicted scores, respectively
precision = dict()
recall = dict()
for i in range(n_classes):
    precision[i], recall[i], _ = precision_recall_curve(y_true == i, y_score[:, i])

# Plot the Precision-Recall Curve for each class
for i in range(n_classes):
    plt.plot(recall[i], precision[i], label=f'Class {i}')

plt.xlabel('Recall')
plt.ylabel('Precision')
plt.title('Precision-Recall Curve for Multi-Class Classification')
plt.legend()
plt.show()

Precision-Recall Curve for Imbalanced Datasets

The Precision-Recall Curve is particularly useful for evaluating classifiers on imbalanced datasets, where the class distributions are skewed. In such cases, the Precision-Recall Curve provides a more informative perspective than the Receiver Operating Characteristic (ROC) curve.

Code:

from sklearn.metrics import precision_recall_curve
import matplotlib.pyplot as plt

# Assuming y_true and y_score are the actual labels and predicted scores for an imbalanced dataset, respectively
precision, recall, _ = precision_recall_curve(y_true, y_score)

plt.plot(recall, precision)
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.title('Precision-Recall Curve for Imbalanced Dataset')
plt.show()

Adjusting for Class Imbalance

When dealing with imbalanced datasets, it is often necessary to adjust the classification threshold to account for the class imbalance. The Precision-Recall Curve can help in selecting an appropriate threshold that balances precision and recall for the minority class.

Code:

from sklearn.metrics import precision_recall_curve
import matplotlib.pyplot as plt

# Assuming y_true and y_score are the actual labels and predicted scores for an imbalanced dataset, respectively
precision, recall, thresholds = precision_recall_curve(y_true, y_score)

# Choose a desired recall value for the minority class
desired_recall = 0.8
threshold_indices = recall >= desired_recall
chosen_threshold = thresholds[threshold_indices][0]
chosen_precision = precision[threshold_indices][0]

print(f"At threshold {chosen_threshold:.2f}, recall is {desired_recall:.2f} and precision is {chosen_precision:.2f}")

Precision-Recall Curve for Anomaly Detection

Anomaly detection is a common use case for the Precision-Recall Curve. In this scenario, the positive class represents anomalies, and the negative class represents normal instances. The Precision-Recall Curve helps evaluate the trade-off between detecting true anomalies and minimizing false positives.

Code:

from sklearn.metrics import precision_recall_curve
import matplotlib.pyplot as plt

# Assuming y_true and y_score are the actual labels and predicted anomaly scores, respectively
precision, recall, _ = precision_recall_curve(y_true, y_score)

plt.plot(recall, precision)
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.title('Precision-Recall Curve for Anomaly Detection')
plt.show()

Precision-Recall Curve for Information Retrieval

In information retrieval tasks, such as document classification or web search, the Precision-Recall Curve is commonly used to evaluate the performance of retrieval systems. High precision means retrieving mostly relevant documents, while high recall means retrieving most of the relevant documents.

Code:

from sklearn.metrics import precision_recall_curve
import matplotlib.pyplot as plt

# Assuming y_true and y_score are the actual labels and predicted relevance scores, respectively
precision, recall, _ = precision_recall_curve(y_true, y_score)

plt.plot(recall, precision)
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.title('Precision-Recall Curve for Information Retrieval')
plt.show()

Additional Resources

For further reading and exploration of the Precision-Recall Curve and related topics, here are some recommended resources from arXiv.org:

  1. "Precision-Recall Curves for Machine Learning Applications" by Saito and Rehmsmeier (arXiv:1508.04697)
  2. "Visualizing the Performance of Binary Classifiers" by Flach (arXiv:1008.2908)
  3. "On the Evaluation of Anomaly Detection Algorithms" by Goix (arXiv:1601.01876)

These resources provide in-depth discussions, mathematical foundations, and advanced techniques related to the Precision-Recall Curve and its applications.

More in Model evaluation and validation

  1. K-Fold Cross Validation with Python10 min

    K-Fold Cross Validation is a statistical method used to estimate the performance of machine learning models. It involves dividing the dataset into K subsets...

  2. Explaining ROC Curve and AUC in Model Evaluation with Python14 min

    The Receiver Operating Characteristic (ROC) curve and Area Under the Curve (AUC) are essential tools for evaluating binary classification models. They help...

  3. Leave-One-Out Cross-Validation with Python13 min

    Leave-One-Out Cross-Validation (LOOCV) is a special case of k-fold cross-validation where k equals the number of data points in the dataset. This technique...

  4. Explaining Leave-One-Out Cross-Validation with Python14 min

    Leave-One-Out Cross-Validation is a technique used in machine learning to assess model performance and generalization. It's a special case of k-fold...

  5. Limitations of R-squared in Regression Model Evaluation:12 min

    Understanding R-squared (R²) in Regression Analysis

  6. Explaining K-Fold Cross-Validation with Python11 min

    K-Fold Cross-Validation is a statistical method used to evaluate machine learning models. It helps assess how well a model will generalize to an independent...