xbe.at

Machine learning / Feature engineering and preprocessing

Diving into Train, Validation, and Test Data in Machine Learning with Python

7 min read, 14 sections

Contents
  1. Understanding Train, Validation, and Test Data
  2. Training Data: The Foundation of Learning
  3. Validation Data: Fine-tuning the Model
  4. Test Data: The Final Evaluation
  5. Cross-Validation: A Robust Evaluation Technique
  6. Stratified Sampling: Maintaining Class Distribution
  7. Time Series Data: Special Considerations
  8. Data Leakage: A Common Pitfall
  9. Feature Selection: When to Split
  10. Handling Imbalanced Datasets
  11. Real-Life Example: Iris Flower Classification
  12. Real-Life Example: Handwritten Digit Recognition
  13. Visualizing Data Splits
  14. Additional Resources

Understanding Train, Validation, and Test Data

In machine learning, the division of data into train, validation, and test sets is crucial for developing robust models. This process helps in training the model, tuning hyperparameters, and evaluating performance on unseen data. Let's explore these concepts using Python and the popular scikit-learn library.

from sklearn.model_selection import train_test_split
import numpy as np

# Generate sample data
X = np.random.rand(1000, 5)
y = np.random.randint(0, 2, 1000)

# Split data into train+validation and test sets
X_train_val, X_test, y_train_val, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Further split train+validation into train and validation sets
X_train, X_val, y_train, y_val = train_test_split(X_train_val, y_train_val, test_size=0.25, random_state=42)

print(f"Train set shape: {X_train.shape}")
print(f"Validation set shape: {X_val.shape}")
print(f"Test set shape: {X_test.shape}")

Training Data: The Foundation of Learning

Training data is the largest portion of your dataset, used to teach the model patterns and relationships. It's the data on which the model learns and adjusts its parameters. Let's create a simple model and train it on our training data.

from sklearn.linear_model import LogisticRegression

# Initialize and train the model
model = LogisticRegression()
model.fit(X_train, y_train)

# Check training accuracy
train_accuracy = model.score(X_train, y_train)
print(f"Training Accuracy: {train_accuracy:.4f}")

Validation Data: Fine-tuning the Model

Validation data is used to tune hyperparameters and prevent overfitting. It provides an unbiased evaluation of the model's performance during training. Let's use our validation set to evaluate the model and compare it with the training performance.

# Evaluate on validation set
val_accuracy = model.score(X_val, y_val)
print(f"Validation Accuracy: {val_accuracy:.4f}")

# Compare with training accuracy
print(f"Difference (Train - Val): {train_accuracy - val_accuracy:.4f}")

Test Data: The Final Evaluation

Test data is kept separate throughout the training process and is used only for the final evaluation of the model. It simulates real-world, unseen data and provides an unbiased estimate of the model's performance. Let's evaluate our model on the test set.

# Evaluate on test set
test_accuracy = model.score(X_test, y_test)
print(f"Test Accuracy: {test_accuracy:.4f}")

# Compare with validation accuracy
print(f"Difference (Val - Test): {val_accuracy - test_accuracy:.4f}")

Cross-Validation: A Robust Evaluation Technique

Cross-validation is a powerful technique that uses multiple train-validation splits to get a more reliable estimate of model performance. It's particularly useful when you have limited data. Let's implement k-fold cross-validation.

from sklearn.model_selection import cross_val_score

# Perform 5-fold cross-validation
cv_scores = cross_val_score(model, X_train_val, y_train_val, cv=5)

print(f"Cross-validation scores: {cv_scores}")
print(f"Mean CV score: {cv_scores.mean():.4f}")
print(f"Standard deviation: {cv_scores.std():.4f}")

Stratified Sampling: Maintaining Class Distribution

When dealing with imbalanced datasets, it's crucial to maintain the class distribution across train, validation, and test sets. Stratified sampling ensures this balance. Let's see how to implement stratified splitting.

from sklearn.model_selection import StratifiedShuffleSplit

# Create stratified split
sss = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42)

# Perform the split
for train_index, test_index in sss.split(X, y):
    X_train_val, X_test = X[train_index], X[test_index]
    y_train_val, y_test = y[train_index], y[test_index]

print(f"Train+Val set shape: {X_train_val.shape}")
print(f"Test set shape: {X_test.shape}")

# Check class distribution
print(f"Original class distribution: {np.bincount(y) / len(y)}")
print(f"Train+Val class distribution: {np.bincount(y_train_val) / len(y_train_val)}")
print(f"Test class distribution: {np.bincount(y_test) / len(y_test)}")

Time Series Data: Special Considerations

When working with time series data, random splitting can lead to data leakage. Instead, we need to use time-based splitting. Let's create a simple time series dataset and split it appropriately.

import pandas as pd
from sklearn.model_selection import TimeSeriesSplit

# Create a time series dataset
dates = pd.date_range(start='2023-01-01', end='2023-12-31', freq='D')
X = pd.DataFrame({'date': dates, 'feature': np.random.rand(len(dates))})
y = np.random.randint(0, 2, len(dates))

# Initialize TimeSeriesSplit
tscv = TimeSeriesSplit(n_splits=5)

# Perform the split
for train_index, test_index in tscv.split(X):
    X_train, X_test = X.iloc[train_index], X.iloc[test_index]
    y_train, y_test = y[train_index], y[test_index]
    print(f"Train set: {X_train.date.min()} to {X_train.date.max()}")
    print(f"Test set: {X_test.date.min()} to {X_test.date.max()}\n")

Data Leakage: A Common Pitfall

Data leakage occurs when information from outside the training dataset is used to create the model. This can lead to overly optimistic performance estimates. Let's demonstrate a common form of data leakage and how to avoid it.

from sklearn.preprocessing import StandardScaler

# Incorrect approach (leakage)
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2, random_state=42)

# Correct approach (no leakage)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

print("Correct approach: Scaler fitted only on training data")
print(f"Train set mean: {X_train_scaled.mean():.4f}")
print(f"Test set mean: {X_test_scaled.mean():.4f}")

Feature Selection: When to Split

Feature selection is an important step in model development, but it's crucial to perform it only on the training data to avoid leakage. Let's demonstrate the correct way to incorporate feature selection in your workflow.

from sklearn.feature_selection import SelectKBest, f_classif

# Split the data first
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Perform feature selection on training data only
selector = SelectKBest(f_classif, k=3)
X_train_selected = selector.fit_transform(X_train, y_train)
X_test_selected = selector.transform(X_test)

print(f"Original feature count: {X_train.shape[1]}")
print(f"Selected feature count: {X_train_selected.shape[1]}")

Handling Imbalanced Datasets

Imbalanced datasets can lead to biased models. Techniques like oversampling, undersampling, or synthetic data generation can help. Let's explore using SMOTE (Synthetic Minority Over-sampling Technique) to balance our dataset.

from imblearn.over_sampling import SMOTE
from collections import Counter

# Create an imbalanced dataset
X_imbalanced = np.random.rand(1000, 5)
y_imbalanced = np.random.choice([0, 1], size=1000, p=[0.9, 0.1])

print("Original class distribution:", Counter(y_imbalanced))

# Apply SMOTE
smote = SMOTE(random_state=42)
X_balanced, y_balanced = smote.fit_resample(X_imbalanced, y_imbalanced)

print("Balanced class distribution:", Counter(y_balanced))

Real-Life Example: Iris Flower Classification

Let's apply our knowledge to a real-world dataset: the Iris flower classification problem. We'll split the data, train a model, and evaluate its performance.

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.metrics import classification_report

# Load the Iris dataset
iris = load_iris()
X, y = iris.data, iris.target

# Split the data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Train the model
model = SVC(kernel='rbf', random_state=42)
model.fit(X_train, y_train)

# Make predictions
y_pred = model.predict(X_test)

# Evaluate the model
print(classification_report(y_test, y_pred, target_names=iris.target_names))

Real-Life Example: Handwritten Digit Recognition

Another practical application is handwritten digit recognition. We'll use the MNIST dataset, prepare the data, and train a simple neural network.

from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier
from sklearn.metrics import accuracy_score

# Load the digits dataset
digits = load_digits()
X, y = digits.data, digits.target

# Split the data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train the model
model = MLPClassifier(hidden_layer_sizes=(50,), max_iter=10, random_state=42)
model.fit(X_train, y_train)

# Make predictions
y_pred = model.predict(X_test)

# Evaluate the model
accuracy = accuracy_score(y_test, y_pred)
print(f"Model accuracy: {accuracy:.4f}")

Visualizing Data Splits

Visualizing how data is split can provide insights into the distribution across sets. Let's create a simple visualization of our train-validation-test split.

import matplotlib.pyplot as plt

# Generate sample data
X = np.random.rand(1000, 2)
y = np.random.randint(0, 2, 1000)

# Split data
X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.3, random_state=42)
X_val, X_test, y_val, y_test = train_test_split(X_temp, y_temp, test_size=0.5, random_state=42)

# Visualize the split
plt.figure(figsize=(10, 8))
plt.scatter(X_train[:, 0], X_train[:, 1], c='blue', label='Train', alpha=0.7)
plt.scatter(X_val[:, 0], X_val[:, 1], c='green', label='Validation', alpha=0.7)
plt.scatter(X_test[:, 0], X_test[:, 1], c='red', label='Test', alpha=0.7)
plt.title('Visualization of Train-Validation-Test Split')
plt.legend()
plt.show()

Additional Resources

For further exploration of train-validation-test splits and related concepts in machine learning:

  1. "A Survey of Cross-Validation Procedures for Model Selection" by Arlot, S. and Celisse, A. (2010) ArXiv: https://arxiv.org/abs/0907.4728
  2. "An Introduction to Statistical Learning" by James, G., Witten, D., Hastie, T., and Tibshirani, R. Available at: https://www.statlearning.com/
  3. Scikit-learn Documentation on Model Selection https://scikit-learn.org/stable/model_selection.html

These resources provide in-depth discussions on data splitting techniques, cross-validation, and their implications in machine learning.

More in Feature engineering and preprocessing

  1. Feature Encoding in Python for Machine Learning6 min

    Feature encoding is a crucial step in data preprocessing for machine learning. It involves converting categorical variables into a numerical format that...

  2. De-normalized Data for Machine Learning Models in Python8 min

    De-normalized data refers to a data structure where redundant information is intentionally added to improve query performance or simplify data handling. This...

  3. Feature Engineering for Machine Learning in Python11 min

    Feature engineering is a crucial process in machine learning that involves transforming raw data into meaningful features that can improve model performance....

  4. Data Warehousing and Dimensional Modeling for Machine Learning Using Python5 min

    Introduction to Data Warehousing and Dimensional Modeling

  5. Feature Engineering for Machine Learning using python4 min

    Feature engineering is the process of transforming raw data into features that better represent the underlying problem to the predictive models, resulting in...

  6. Data Version Control for Machine Learning with Python7 min

    Data Version Control (DVC) is a powerful tool for managing and versioning datasets and machine learning models. It helps data scientists and ML engineers...