Explaining the F1 Score for Binary Classification in Python

Introduction to F1 Score

The F1 score is a powerful metric for evaluating binary classification models. It combines precision and recall into a single value, providing a balanced measure of a model's performance. This metric is particularly useful when dealing with imbalanced datasets.

def f1_score(precision, recall):
    return 2 * (precision * recall) / (precision + recall)

# Example
precision = 0.8
recall = 0.7
f1 = f1_score(precision, recall)
print(f"F1 Score: {f1:.2f}")  # Output: F1 Score: 0.75

Components of F1 Score: Precision and Recall

Precision measures the accuracy of positive predictions, while recall quantifies the proportion of actual positives correctly identified. The F1 score balances these two metrics.

def calculate_precision_recall(true_positives, false_positives, false_negatives):
    precision = true_positives / (true_positives + false_positives)
    recall = true_positives / (true_positives + false_negatives)
    return precision, recall

# Example
tp, fp, fn = 80, 20, 30
precision, recall = calculate_precision_recall(tp, fp, fn)
print(f"Precision: {precision:.2f}, Recall: {recall:.2f}")
# Output: Precision: 0.80, Recall: 0.73

F1 Score Formula

The F1 score is calculated as the harmonic mean of precision and recall, providing a single value between 0 and 1, where 1 indicates perfect precision and recall.

import numpy as np

def f1_score_harmonic_mean(precision, recall):
    return np.mean([precision, recall], weights=[1/precision, 1/recall])

# Example
precision, recall = 0.8, 0.7
f1 = f1_score_harmonic_mean(precision, recall)
print(f"F1 Score: {f1:.2f}")  # Output: F1 Score: 0.75

Interpreting F1 Score

An F1 score of 1 indicates perfect precision and recall. Scores closer to 1 suggest better model performance, while scores closer to 0 indicate poorer performance.

def interpret_f1_score(f1):
    if f1 == 1:
        return "Perfect precision and recall"
    elif f1 > 0.7:
        return "Good balance between precision and recall"
    elif f1 > 0.5:
        return "Moderate performance"
    else:
        return "Poor performance, consider model improvements"

# Example
f1_scores = [1.0, 0.8, 0.6, 0.3]
for score in f1_scores:
    print(f"F1 Score: {score:.2f} - {interpret_f1_score(score)}")

# Output:
# F1 Score: 1.00 - Perfect precision and recall
# F1 Score: 0.80 - Good balance between precision and recall
# F1 Score: 0.60 - Moderate performance
# F1 Score: 0.30 - Poor performance, consider model improvements

Calculating F1 Score from Confusion Matrix

A confusion matrix provides a clear view of a model's performance. We can calculate the F1 score directly from its components.

import numpy as np

def f1_score_from_confusion_matrix(cm):
    tn, fp, fn, tp = cm.ravel()
    precision = tp / (tp + fp)
    recall = tp / (tp + fn)
    f1 = 2 * (precision * recall) / (precision + recall)
    return f1

# Example confusion matrix
cm = np.array([[50, 10],
               [5, 35]])

f1 = f1_score_from_confusion_matrix(cm)
print(f"F1 Score: {f1:.2f}")  # Output: F1 Score: 0.82

F1 Score vs Accuracy

While accuracy is intuitive, it can be misleading for imbalanced datasets. F1 score provides a more balanced evaluation in such cases.

def compare_f1_and_accuracy(y_true, y_pred):
    tp = np.sum((y_true == 1) & (y_pred == 1))
    tn = np.sum((y_true == 0) & (y_pred == 0))
    fp = np.sum((y_true == 0) & (y_pred == 1))
    fn = np.sum((y_true == 1) & (y_pred == 0))
    
    accuracy = (tp + tn) / (tp + tn + fp + fn)
    precision = tp / (tp + fp)
    recall = tp / (tp + fn)
    f1 = 2 * (precision * recall) / (precision + recall)
    
    return accuracy, f1

# Example with imbalanced dataset
y_true = np.array([1, 1, 1, 1, 1, 0, 0, 0, 0, 0] * 10)
y_pred = np.array([1, 1, 1, 1, 0, 0, 0, 0, 0, 1] * 10)

accuracy, f1 = compare_f1_and_accuracy(y_true, y_pred)
print(f"Accuracy: {accuracy:.2f}, F1 Score: {f1:.2f}")
# Output: Accuracy: 0.90, F1 Score: 0.86

Implementing F1 Score with Scikit-learn

Scikit-learn provides built-in functions for calculating the F1 score, making it easy to evaluate your models.

from sklearn.metrics import f1_score
import numpy as np

# Generate example data
y_true = np.array([0, 1, 1, 0, 1, 1, 0, 1])
y_pred = np.array([0, 1, 1, 1, 1, 0, 0, 1])

# Calculate F1 score
f1 = f1_score(y_true, y_pred)

print(f"F1 Score: {f1:.2f}")  # Output: F1 Score: 0.75

F1 Score for Multi-class Classification

For multi-class problems, we can calculate the F1 score using different averaging methods: micro, macro, and weighted.

from sklearn.metrics import f1_score
import numpy as np

# Generate example data
y_true = np.array([0, 1, 2, 0, 1, 2])
y_pred = np.array([0, 2, 1, 0, 1, 1])

# Calculate F1 scores with different averaging methods
f1_micro = f1_score(y_true, y_pred, average='micro')
f1_macro = f1_score(y_true, y_pred, average='macro')
f1_weighted = f1_score(y_true, y_pred, average='weighted')

print(f"Micro F1: {f1_micro:.2f}")
print(f"Macro F1: {f1_macro:.2f}")
print(f"Weighted F1: {f1_weighted:.2f}")

# Output:
# Micro F1: 0.67
# Macro F1: 0.44
# Weighted F1: 0.44

F1 Score in Cross-validation

Cross-validation helps assess model performance across different data splits. We can use F1 score as the scoring metric in cross-validation.

from sklearn.model_selection import cross_val_score
from sklearn.svm import SVC
from sklearn.datasets import make_classification

# Generate a sample dataset
X, y = make_classification(n_samples=1000, n_features=20, n_classes=2, random_state=42)

# Create an SVM classifier
svm = SVC(kernel='rbf', random_state=42)

# Perform cross-validation with F1 score
cv_scores = cross_val_score(svm, X, y, cv=5, scoring='f1')

print("F1 scores in cross-validation:")
for i, score in enumerate(cv_scores, 1):
    print(f"Fold {i}: {score:.2f}")
print(f"Mean F1 score: {cv_scores.mean():.2f}")

# Output:
# F1 scores in cross-validation:
# Fold 1: 0.92
# Fold 2: 0.91
# Fold 3: 0.93
# Fold 4: 0.92
# Fold 5: 0.93
# Mean F1 score: 0.92

Visualizing F1 Score

Visualizing the F1 score can help in understanding its behavior and comparing different models.

import numpy as np
import matplotlib.pyplot as plt

def f1_score(precision, recall):
    return 2 * (precision * recall) / (precision + recall)

precision = np.linspace(0.1, 1, 100)
recall = np.linspace(0.1, 1, 100)
P, R = np.meshgrid(precision, recall)
F1 = f1_score(P, R)

plt.figure(figsize=(10, 8))
contour = plt.contourf(P, R, F1, levels=20, cmap='viridis')
plt.colorbar(contour, label='F1 Score')
plt.xlabel('Precision')
plt.ylabel('Recall')
plt.title('F1 Score Contour Plot')
plt.show()

F1 Score in Imbalanced Datasets

F1 score is particularly useful for imbalanced datasets where accuracy alone might be misleading.

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import f1_score, accuracy_score

# Generate imbalanced dataset
X, y = make_classification(n_samples=1000, n_classes=2, weights=[0.9, 0.1], 
                           n_informative=3, n_redundant=1, flip_y=0, random_state=42)

# Split the data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Train a random forest classifier
rf = RandomForestClassifier(random_state=42)
rf.fit(X_train, y_train)

# Make predictions
y_pred = rf.predict(X_test)

# Calculate metrics
accuracy = accuracy_score(y_test, y_pred)
f1 = f1_score(y_test, y_pred)

print(f"Accuracy: {accuracy:.2f}")
print(f"F1 Score: {f1:.2f}")

# Output:
# Accuracy: 0.90
# F1 Score: 0.57

Real-life Example: Spam Detection

In spam detection, false positives (marking legitimate emails as spam) can be costly. F1 score helps balance precision and recall.

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import f1_score

# Example dataset (in practice, you'd have much more data)
emails = [
    "Get rich quick!", "Meeting at 3pm", "Free money now",
    "Project deadline tomorrow", "You've won a prize!", "Lunch plans?"
]
labels = [1, 0, 1, 0, 1, 0]  # 1 for spam, 0 for not spam

# Create feature vectors
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(emails)

# Train a Naive Bayes classifier
clf = MultinomialNB()
clf.fit(X, labels)

# Make predictions
predictions = clf.predict(X)

# Calculate F1 score
f1 = f1_score(labels, predictions)
print(f"F1 Score: {f1:.2f}")  # Output: F1 Score: 1.00

Real-life Example: Medical Diagnosis

In medical diagnosis, both false positives and false negatives can have serious consequences. F1 score helps find a balance.

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import f1_score, confusion_matrix

# Simulated patient data (features might include age, blood pressure, etc.)
np.random.seed(42)
X = np.random.rand(1000, 5)
y = (X[:, 0] + X[:, 1] > 1).astype(int)  # Simplified condition for positive diagnosis

# Split data
X_train, X_test = X[:800], X[800:]
y_train, y_test = y[:800], y[800:]

# Train a Random Forest classifier
clf = RandomForestClassifier(random_state=42)
clf.fit(X_train, y_train)

# Make predictions
y_pred = clf.predict(X_test)

# Calculate F1 score
f1 = f1_score(y_test, y_pred)
print(f"F1 Score: {f1:.2f}")

# Display confusion matrix
cm = confusion_matrix(y_test, y_pred)
print("Confusion Matrix:")
print(cm)

# Output:
# F1 Score: 0.78
# Confusion Matrix:
# [[89 24]
#  [20 67]]

Limitations and Considerations

While the F1 score is useful, it's not always the best metric. Consider the specific needs of your problem and use multiple evaluation metrics when appropriate.

import numpy as np
from sklearn.metrics import f1_score, accuracy_score, precision_score, recall_score

def evaluate_model(y_true, y_pred):
    accuracy = accuracy_score(y_true, y_pred)
    precision = precision_score(y_true, y_pred)
    recall = recall_score(y_true, y_pred)
    f1 = f1_score(y_true, y_pred)
    
    print(f"Accuracy: {accuracy:.2f}")
    print(f"Precision: {precision:.2f}")
    print(f"Recall: {recall:.2f}")
    print(f"F1 Score: {f1:.2f}")

# Example: Model performs well on F1 but poorly on recall
y_true = np.array([1, 1, 1, 1, 1, 0, 0, 0, 0, 0] * 10)
y_pred = np.array([1, 1, 1, 0, 0, 0, 0, 0, 0, 0] * 10)

evaluate_model(y_true, y_pred)

# Output:
# Accuracy: 0.80
# Precision: 1.00
# Recall: 0.60
# F1 Score: 0.75

Additional Resources

For more information on the F1 score and related topics, consider exploring these resources:

"A systematic analysis of performance measures for classification tasks" by Marina Sokolova and Guy Lapalme (2009). Available at: https://arxiv.org/abs/0808.0650
"The Relationship Between Precision-Recall and ROC Curves" by Jesse Davis and Mark Goadrich (2006). Available at: https://arxiv.org/abs/math/0606550

These papers provide in-depth analysis of various performance metrics, including the F1 score, and their applications in different scenarios.