xbe.at

Deep learning / Activation functions and loss

Cross-Entropy Loss for Multiclass Classification

5 min read, 15 sections

Contents
  1. Cross-Entropy Loss in Classification Models
  2. Source Code for Cross-Entropy Loss in Classification Models
  3. Ordinal Datasets and Their Challenges
  4. Source Code for Ordinal Datasets and Their Challenges
  5. Ranking Inconsistencies
  6. Source Code for Ranking Inconsistencies
  7. Cumulative Probability Interpretation
  8. Source Code for Cumulative Probability Interpretation
  9. Building a Rank-Consistent Classifier
  10. Source Code for Building a Rank-Consistent Classifier
  11. Real-Life Example: Academic Performance Prediction
  12. Source Code for Academic Performance Prediction
  13. Real-Life Example: Customer Satisfaction Surveys
  14. Source Code for Customer Satisfaction Surveys
  15. Additional Resources

Cross-Entropy Loss in Classification Models

Cross-entropy loss is widely used for training neural networks in classification tasks. It measures the dissimilarity between predicted probability distributions and true labels. However, this loss function has limitations when dealing with ordinal datasets, where class labels have a natural order.

Source Code for Cross-Entropy Loss in Classification Models

import math

def cross_entropy_loss(y_true, y_pred):
    """
    Calculate cross-entropy loss for a single sample
    y_true: one-hot encoded true label
    y_pred: predicted probabilities for each class
    """
    epsilon = 1e-15  # Small value to avoid log(0)
    return -sum(y_true[i] * math.log(y_pred[i] + epsilon) for i in range(len(y_true)))

# Example usage
y_true = [0, 1, 0, 0]  # One-hot encoded true label (class 1)
y_pred = [0.1, 0.6, 0.2, 0.1]  # Predicted probabilities

loss = cross_entropy_loss(y_true, y_pred)
print(f"Cross-entropy loss: {loss:.4f}")

Ordinal Datasets and Their Challenges

Ordinal datasets contain classes with a natural order, such as age groups (child, teenager, young adult, middle-aged, senior). Cross-entropy loss treats these classes as independent categories, ignoring their inherent ordering. This can lead to ranking inconsistencies in the model's predictions.

Source Code for Ordinal Datasets and Their Challenges

def simulate_age_group_prediction(true_age_group, predicted_probabilities):
    age_groups = ["child", "teenager", "young adult", "middle-aged", "senior"]
    
    print(f"True age group: {age_groups[true_age_group]}")
    print("Predicted probabilities:")
    for group, prob in zip(age_groups, predicted_probabilities):
        print(f"{group}: {prob:.2f}")
    
    predicted_group = age_groups[predicted_probabilities.index(max(predicted_probabilities))]
    print(f"Predicted age group: {predicted_group}")

# Example with ranking inconsistency
true_age_group = 2  # young adult
predicted_probabilities = [0.1, 0.3, 0.2, 0.3, 0.1]

simulate_age_group_prediction(true_age_group, predicted_probabilities)

Ranking Inconsistencies

Ranking inconsistencies occur when predicted probabilities for adjacent labels don't align with their natural order. For example, if a model predicts a lower probability for "child" than "teenager," it contradicts the logical age progression. This issue stems from cross-entropy's treatment of each class as a separate, unordered category.

Source Code for Ranking Inconsistencies

def check_ranking_consistency(probabilities):
    """
    Check if probabilities are consistent with ordinal ranking
    """
    return all(probabilities[i] >= probabilities[i+1] for i in range(len(probabilities)-1))

# Example 1: Inconsistent ranking
inconsistent_probs = [0.1, 0.3, 0.2, 0.3, 0.1]
print("Inconsistent probabilities:", inconsistent_probs)
print("Is consistent?", check_ranking_consistency(inconsistent_probs))

# Example 2: Consistent ranking
consistent_probs = [0.4, 0.3, 0.2, 0.1, 0.0]
print("\nConsistent probabilities:", consistent_probs)
print("Is consistent?", check_ranking_consistency(consistent_probs))

Cumulative Probability Interpretation

A more suitable approach for ordinal datasets is to interpret predictions as cumulative probabilities. For instance, if the true label is "young adult," we want the model to indicate that the input is "at least a child," "at least a teenager," and "at least a young adult."

Source Code for Cumulative Probability Interpretation

def cumulative_probabilities(probabilities):
    """
    Convert individual probabilities to cumulative probabilities
    """
    return [sum(probabilities[:i+1]) for i in range(len(probabilities))]

def interpret_cumulative_probs(cum_probs):
    age_groups = ["child", "teenager", "young adult", "middle-aged", "senior"]
    for group, prob in zip(age_groups, cum_probs):
        print(f"Probability of being at least a {group}: {prob:.2f}")

# Example
individual_probs = [0.2, 0.3, 0.3, 0.1, 0.1]
cum_probs = cumulative_probabilities(individual_probs)

print("Individual probabilities:", individual_probs)
print("Cumulative probabilities:", cum_probs)
print("\nInterpretation:")
interpret_cumulative_probs(cum_probs)

Building a Rank-Consistent Classifier

To address the limitations of cross-entropy loss for ordinal datasets, we can build a rank-consistent classifier. This approach ensures that the model learns and generalizes the correct progression of ordinal classes, such as age groups.

Source Code for Building a Rank-Consistent Classifier

import math

def ordinal_loss(y_true, y_pred):
    """
    Calculate ordinal regression loss
    y_true: true ordinal label (integer)
    y_pred: predicted cumulative probabilities
    """
    loss = 0
    for k in range(len(y_pred)):
        if k <= y_true:
            loss += -math.log(y_pred[k])
        else:
            loss += -math.log(1 - y_pred[k])
    return loss

# Example usage
y_true = 2  # young adult
y_pred = [0.9, 0.7, 0.5, 0.2, 0.1]  # Predicted cumulative probabilities

loss = ordinal_loss(y_true, y_pred)
print(f"Ordinal regression loss: {loss:.4f}")

Real-Life Example: Academic Performance Prediction

Consider predicting a student's academic performance level (poor, average, good, excellent) based on various factors. This is an ordinal classification problem where the performance levels have a natural order.

Source Code for Academic Performance Prediction

def predict_academic_performance(study_hours, attendance, previous_grades):
    # Simple ordinal classifier (for illustration purposes)
    score = study_hours * 0.4 + attendance * 0.3 + previous_grades * 0.3
    
    if score < 0.3:
        return [0.7, 0.2, 0.1, 0.0]  # Poor
    elif score < 0.6:
        return [0.9, 0.7, 0.2, 0.1]  # Average
    elif score < 0.8:
        return [1.0, 0.9, 0.7, 0.2]  # Good
    else:
        return [1.0, 1.0, 0.9, 0.8]  # Excellent

# Example usage
study_hours = 6  # hours per day
attendance = 0.9  # 90% attendance
previous_grades = 0.75  # 75/100 average

prediction = predict_academic_performance(study_hours, attendance, previous_grades)
performance_levels = ["Poor", "Average", "Good", "Excellent"]

print("Predicted cumulative probabilities:")
for level, prob in zip(performance_levels, prediction):
    print(f"{level}: {prob:.2f}")

predicted_level = performance_levels[sum(p > 0.5 for p in prediction) - 1]
print(f"\nPredicted performance level: {predicted_level}")

Real-Life Example: Customer Satisfaction Surveys

Customer satisfaction surveys often use ordinal scales (e.g., very unsatisfied, unsatisfied, neutral, satisfied, very satisfied). A rank-consistent classifier can better model these responses, respecting the inherent order of satisfaction levels.

Source Code for Customer Satisfaction Surveys

def predict_customer_satisfaction(product_quality, service_quality, price_satisfaction):
    # Simple ordinal classifier for customer satisfaction
    score = product_quality * 0.4 + service_quality * 0.4 + price_satisfaction * 0.2
    
    if score < 0.2:
        return [0.8, 0.2, 0.1, 0.0, 0.0]  # Very Unsatisfied
    elif score < 0.4:
        return [1.0, 0.7, 0.2, 0.1, 0.0]  # Unsatisfied
    elif score < 0.6:
        return [1.0, 0.9, 0.7, 0.3, 0.1]  # Neutral
    elif score < 0.8:
        return [1.0, 1.0, 0.9, 0.7, 0.3]  # Satisfied
    else:
        return [1.0, 1.0, 1.0, 0.9, 0.8]  # Very Satisfied

# Example usage
product_quality = 0.8  # 80/100
service_quality = 0.7  # 70/100
price_satisfaction = 0.6  # 60/100

prediction = predict_customer_satisfaction(product_quality, service_quality, price_satisfaction)
satisfaction_levels = ["Very Unsatisfied", "Unsatisfied", "Neutral", "Satisfied", "Very Satisfied"]

print("Predicted cumulative probabilities:")
for level, prob in zip(satisfaction_levels, prediction):
    print(f"{level}: {prob:.2f}")

predicted_level = satisfaction_levels[sum(p > 0.5 for p in prediction) - 1]
print(f"\nPredicted satisfaction level: {predicted_level}")

Additional Resources

For more information on ordinal classification and rank-consistent models, refer to the following research papers:

  1. Niu, Z., Zhou, M., Wang, L., Gao, X., & Hua, G. (2016). Ordinal Regression with Multiple Output CNN for Age Estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). arXiv:1511.06664
  2. Cheng, J., Wang, Z., & Pollastri, G. (2008). A neural network approach to ordinal regression. In 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence). arXiv:0704.1028
  3. Gutiérrez, P. A., Pérez-Ortiz, M., Sánchez-Monedero, J., Fernández-Navarro, F., & Hervás-Martínez, C. (2016). Ordinal regression methods: survey and experimental study. IEEE Transactions on Knowledge and Data Engineering, 28(1), 127-146. arXiv:1503.07292

These papers provide in-depth discussions on various approaches to ordinal classification, including neural network-based methods and comparative studies of different ordinal regression techniques.

More in Activation functions and loss

  1. Discover Activation Functions in Neural Networks9 min

    The Rectified Linear Unit (ReLU) is one of the most widely used activation functions in modern neural networks. It transforms input by outputting zero for...

  2. Common Loss Functions for Deep Learning Using Python4 min

    In deep learning, loss functions play a crucial role in training models by quantifying the difference between the predicted output and the actual target....

  3. Essentials of Neural Network Activation Functions in Python5 min

    Activation functions are crucial components in neural networks, determining the output of a neuron given an input or set of inputs. They introduce...

  4. Building a Softmax Activation Function from Scratch in Python7 min

    The Softmax function is a crucial component in many machine learning models, particularly in multi-class classification problems. It transforms a vector of...

  5. Evaluating Model Performance with Loss Functions10 min

    Loss functions are fundamental components in machine learning that measure how well a model's predictions match the actual values. They quantify the...

  6. Building a Sigmoid Activation Function from Scratch in Python7 min

    The sigmoid function is a crucial component in neural networks, particularly in binary classification problems. It maps any input value to a number between 0...