Feature Scaling in Machine Learning Algorithms

Introduction to Feature Scaling in Machine Learning

Feature scaling is a crucial preprocessing step in many machine learning algorithms. It involves transforming the features to a common scale, which can significantly improve the performance and convergence of certain algorithms. This presentation will explore the concept of feature scaling, its importance, common methods, and when to apply them.

import matplotlib.pyplot as plt

# Generate sample data
np.random.seed(42)
feature1 = np.random.normal(0, 1, 100)
feature2 = np.random.normal(0, 100, 100)

# Plot original data
plt.figure(figsize=(10, 5))
plt.subplot(121)
plt.scatter(feature1, feature2)
plt.title('Original Data')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')

plt.show()

Why Feature Scaling Matters

Feature scaling is essential when dealing with features of different scales. Many machine learning algorithms, such as gradient descent-based optimization and distance-based methods, are sensitive to the scale of input features. Unscaled features can lead to:

Slow convergence in optimization algorithms
Biased importance of features in distance-based methods
Numerical instability in some algorithms

def gradient_descent(X, y, learning_rate, num_iterations):
    m, n = X.shape
    theta = np.zeros(n)
    
    for _ in range(num_iterations):
        h = np.dot(X, theta)
        gradient = np.dot(X.T, (h - y)) / m
        theta -= learning_rate * gradient
    
    return theta

# Generate sample data
X = np.column_stack((feature1, feature2))
y = 2 * feature1 + 0.5 * feature2 + np.random.normal(0, 0.1, 100)

# Run gradient descent
theta = gradient_descent(X, y, learning_rate=0.01, num_iterations=1000)
print("Estimated coefficients:", theta)

Standardization: Z-Score Normalization

Standardization, also known as Z-score normalization, transforms features to have zero mean and unit variance. It's calculated as:

z = \frac{x - \mu}{\sigma}

Where x is the original feature value, \mu is the mean of the feature, and \sigma is the standard deviation.


# Standardize features
scaler = StandardScaler()
X_standardized = scaler.fit_transform(X)

print("Original mean:", X.mean(axis=0))
print("Original std:", X.std(axis=0))
print("Standardized mean:", X_standardized.mean(axis=0))
print("Standardized std:", X_standardized.std(axis=0))

# Plot standardized data
plt.figure(figsize=(10, 5))
plt.scatter(X_standardized[:, 0], X_standardized[:, 1])
plt.title('Standardized Data')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.show()

Min-Max Normalization

Min-Max normalization scales features to a fixed range, typically [0, 1]. The formula is:

x_{normalized} = \frac{x - x_{min}}{x_{max} - x_{min}}

Where x is the original value, and x_{min} and x_{max} are the minimum and maximum values of the feature.


# Apply Min-Max scaling
scaler = MinMaxScaler()
X_normalized = scaler.fit_transform(X)

print("Original min:", X.min(axis=0))
print("Original max:", X.max(axis=0))
print("Normalized min:", X_normalized.min(axis=0))
print("Normalized max:", X_normalized.max(axis=0))

# Plot normalized data
plt.figure(figsize=(10, 5))
plt.scatter(X_normalized[:, 0], X_normalized[:, 1])
plt.title('Min-Max Normalized Data')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.show()

When to Use Standardization

Standardization is preferred when:

The algorithm assumes normally distributed data (e.g., linear regression, logistic regression)
You want to preserve zero as a meaningful value
The features have significantly different variances
You're using algorithms sensitive to the magnitude of features (e.g., neural networks)

from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Generate binary classification data
y_binary = (y > y.mean()).astype(int)

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y_binary, test_size=0.2, random_state=42)

# Train without standardization
lr = LogisticRegression(random_state=42)
lr.fit(X_train, y_train)
y_pred = lr.predict(X_test)
print("Accuracy without standardization:", accuracy_score(y_test, y_pred))

# Train with standardization
scaler = StandardScaler()
X_train_std = scaler.fit_transform(X_train)
X_test_std = scaler.transform(X_test)

lr_std = LogisticRegression(random_state=42)
lr_std.fit(X_train_std, y_train)
y_pred_std = lr_std.predict(X_test_std)
print("Accuracy with standardization:", accuracy_score(y_test, y_pred_std))

When to Use Min-Max Normalization

Min-Max normalization is suitable when:

You need features on a fixed scale, like [0, 1]
The distribution of the data is not Gaussian or unknown
You're using algorithms that require non-negative values (e.g., neural networks with sigmoid activation)
Preserving zero values in sparse data is important

from sklearn.preprocessing import MinMaxScaler

# Generate sample data
X = np.random.rand(1000, 5) * 100
y = (X[:, 0] + X[:, 1] - X[:, 2] + X[:, 3] - X[:, 4] > 0).astype(int)

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Normalize data
scaler = MinMaxScaler()
X_train_norm = scaler.fit_transform(X_train)
X_test_norm = scaler.transform(X_test)

# Create and train model
model = tf.keras.Sequential([
    tf.keras.layers.Dense(10, activation='relu', input_shape=(5,)),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
model.fit(X_train_norm, y_train, epochs=10, batch_size=32, validation_split=0.2, verbose=0)

# Evaluate model
_, accuracy = model.evaluate(X_test_norm, y_test, verbose=0)
print(f"Model accuracy with Min-Max normalization: {accuracy:.4f}")

Robust Scaling

Robust scaling is useful when dealing with datasets containing outliers. It uses the median and interquartile range instead of mean and standard deviation:

x_{scaled} = \frac{x - median(x)}{IQR(x)}

Where IQR(x) is the interquartile range of the feature.


# Generate data with outliers
X = np.random.randn(100, 2)
X[0] = [100, 100]  # Add an outlier

# Apply robust scaling
scaler = RobustScaler()
X_robust = scaler.fit_transform(X)

# Plot original and scaled data
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5))

ax1.scatter(X[:, 0], X[:, 1])
ax1.set_title('Original Data with Outlier')

ax2.scatter(X_robust[:, 0], X_robust[:, 1])
ax2.set_title('Robust Scaled Data')

plt.show()

Feature Scaling in Practice: Preprocessing Pipeline

In real-world scenarios, it's crucial to apply feature scaling consistently across training and test sets. Scikit-learn's Pipeline class helps create a preprocessing workflow that can be easily applied to new data.

from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

# Generate a binary classification dataset
X, y = make_classification(n_samples=1000, n_features=20, random_state=42)

# Split the data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Create a pipeline
pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('svm', SVC(kernel='rbf'))
])

# Fit the pipeline
pipeline.fit(X_train, y_train)

# Evaluate the model
train_score = pipeline.score(X_train, y_train)
test_score = pipeline.score(X_test, y_test)

print(f"Training accuracy: {train_score:.4f}")
print(f"Test accuracy: {test_score:.4f}")

Real-Life Example: Image Recognition

In image recognition tasks, feature scaling is often applied to pixel values to normalize them to a common range. This helps in faster convergence during training and can improve model performance.

from tensorflow.keras.datasets import mnist
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Flatten, Dense

# Load and preprocess MNIST dataset
(x_train, y_train), (x_test, y_test) = mnist.load_data()

# Normalize pixel values to [0, 1]
x_train = x_train.astype('float32') / 255
x_test = x_test.astype('float32') / 255

# Create and train model
model = Sequential([
    Flatten(input_shape=(28, 28)),
    Dense(128, activation='relu'),
    Dense(10, activation='softmax')
])

model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
model.fit(x_train, y_train, epochs=5, validation_split=0.1, verbose=0)

# Evaluate model
_, accuracy = model.evaluate(x_test, y_test, verbose=0)
print(f"Test accuracy: {accuracy:.4f}")

Real-Life Example: Customer Churn Prediction

In customer churn prediction, features often have different scales (e.g., age, income, number of transactions). Proper scaling ensures that all features contribute equally to the model.

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, classification_report

# Generate sample customer data
np.random.seed(42)
n_samples = 1000

data = pd.DataFrame({
    'age': np.random.randint(18, 80, n_samples),
    'income': np.random.randint(20000, 200000, n_samples),
    'transactions': np.random.randint(1, 100, n_samples),
    'customer_service_calls': np.random.randint(0, 10, n_samples)
})

# Generate churn labels (simplified for demonstration)
data['churn'] = ((data['age'] > 60) & (data['transactions'] < 10) | 
                 (data['income'] < 50000) & (data['customer_service_calls'] > 5)).astype(int)

# Split data
X = data.drop('churn', axis=1)
y = data['churn']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Scale features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Train and evaluate model
rf = RandomForestClassifier(random_state=42)
rf.fit(X_train_scaled, y_train)
y_pred = rf.predict(X_test_scaled)

print("Accuracy:", accuracy_score(y_test, y_pred))
print("\nClassification Report:")
print(classification_report(y_test, y_pred))

Handling Mixed Data Types

In real-world datasets, you often encounter a mix of numerical and categorical features. It's important to handle each type appropriately during the scaling process.

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.pipeline import Pipeline
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

# Create a sample dataset with mixed types
data = pd.DataFrame({
    'age': np.random.randint(18, 80, 1000),
    'income': np.random.randint(20000, 200000, 1000),
    'education': np.random.choice(['High School', 'Bachelor', 'Master', 'PhD'], 1000),
    'occupation': np.random.choice(['Engineer', 'Teacher', 'Doctor', 'Artist'], 1000),
    'churn': np.random.choice([0, 1], 1000)
})

# Split features and target
X = data.drop('churn', axis=1)
y = data['churn']

# Split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Create preprocessor
preprocessor = ColumnTransformer(
    transformers=[
        ('num', StandardScaler(), ['age', 'income']),
        ('cat', OneHotEncoder(drop='first'), ['education', 'occupation'])
    ])

# Create a pipeline
pipeline = Pipeline([
    ('preprocessor', preprocessor),
    ('classifier', RandomForestClassifier(random_state=42))
])

# Fit the pipeline
pipeline.fit(X_train, y_train)

# Evaluate the model
train_score = pipeline.score(X_train, y_train)
test_score = pipeline.score(X_test, y_test)

print(f"Training accuracy: {train_score:.4f}")
print(f"Test accuracy: {test_score:.4f}")

Feature Scaling and Model Interpretability

While feature scaling is crucial for many algorithms, it can affect model interpretability, especially for linear models. Standardization preserves the relative importance of features, but the coefficients are no longer in the original scale.

from sklearn.preprocessing import StandardScaler

# Generate sample data
np.random.seed(42)
X = np.random.randn(1000, 3)
y = (2*X[:, 0] + 0.5*X[:, 1] - X[:, 2] > 0).astype(int)

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train model without scaling
lr = LogisticRegression(random_state=42)
lr.fit(X_train, y_train)
print("Coefficients without scaling:", lr.coef_[0])

# Scale features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Train model with scaled features
lr_scaled = LogisticRegression(random_state=42)
lr_scaled.fit(X_train_scaled, y_train)
print("Coefficients with scaling:", lr_scaled.coef_[0])

# Compare feature importances
importances = np.abs(lr.coef_[0])
scaled_importances = np.abs(lr_scaled.coef_[0])

print("\nFeature importances:")
for i in range(3):
    print(f"Feature {i+1}: Original = {importances[i]:.4f}, Scaled = {scaled_importances[i]:.4f}")

Scaling in Time Series Data

When dealing with time series data, it's important to consider the temporal nature of the information. One approach is to use a sliding window for scaling to preserve the relative changes over time.

from sklearn.preprocessing import MinMaxScaler

# Generate sample time series data
dates = pd.date_range(start='2023-01-01', end='2023-12-31', freq='D')
values = np.cumsum(np.random.randn(len(dates))) + 100
ts_data = pd.DataFrame({'date': dates, 'value': values})

# Function to scale using a sliding window
def sliding_window_scale(data, window_size=30):
    scaler = MinMaxScaler()
    scaled_data = []
    
    for i in range(len(data)):
        start = max(0, i - window_size + 1)
        window = data[start:i+1]
        scaled_value = scaler.fit_transform(window.reshape(-1, 1))[-1][0]
        scaled_data.append(scaled_value)
    
    return np.array(scaled_data)

# Apply sliding window scaling
ts_data['scaled_value'] = sliding_window_scale(ts_data['value'].values)

# Plot original and scaled data
plt.figure(figsize=(12, 6))
plt.plot(ts_data['date'], ts_data['value'], label='Original')
plt.plot(ts_data['date'], ts_data['scaled_value'], label='Scaled')
plt.title('Time Series Data: Original vs Scaled')
plt.legend()
plt.show()

Feature Scaling in Unsupervised Learning

Feature scaling is particularly important in unsupervised learning algorithms like clustering, where the goal is to find patterns based on similarities or distances between data points.

from sklearn.preprocessing import StandardScaler
import matplotlib.pyplot as plt

# Generate sample data
np.random.seed(42)
X = np.random.randn(300, 2)
X[:100, 0] += 2
X[100:200, 0] -= 2
X[200:, 1] += 3

# Perform K-means clustering without scaling
kmeans = KMeans(n_clusters=3, random_state=42)
clusters = kmeans.fit_predict(X)

# Scale the data
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Perform K-means clustering with scaled data
kmeans_scaled = KMeans(n_clusters=3, random_state=42)
clusters_scaled = kmeans_scaled.fit_predict(X_scaled)

# Plot results
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5))

ax1.scatter(X[:, 0], X[:, 1], c=clusters, cmap='viridis')
ax1.set_title('K-means without Scaling')

ax2.scatter(X_scaled[:, 0], X_scaled[:, 1], c=clusters_scaled, cmap='viridis')
ax2.set_title('K-means with Scaling')

plt.show()

Additional Resources

For those interested in diving deeper into feature scaling and its applications in machine learning, here are some valuable resources:

"Feature Scaling for Machine Learning: Understanding the Difference Between Normalization vs. Standardization" by Sebastian Raschka (arXiv:1411.5754)
"A Survey on Feature Scaling Techniques and Their Applications in Machine Learning" by Jianqing Fan and Qiang Sun (arXiv:1812.04259)
"The Effect of Scaling on the Performance of Classification Algorithms" by Xindong Wu et al. (arXiv:1908.05070)

These papers provide in-depth discussions on various feature scaling techniques, their theoretical foundations, and practical implications in machine learning applications.