xbe.at

Machine learning / Feature engineering and preprocessing

Feature Encoding in Python for Machine Learning

6 min read, 15 sections

Contents
  1. Feature Encoding: An Introduction
  2. One-Hot Encoding
  3. Binary Encoding
  4. Ordinal Encoding
  5. Frequency Encoding
  6. Target Encoding
  7. Feature Hashing
  8. Embedding Encoding
  9. Cyclical Encoding
  10. Real-Life Example: Weather Prediction
  11. Real-Life Example: Text Classification
  12. Handling Missing Values in Categorical Features
  13. Combining Multiple Encoding Techniques
  14. Encoding for Time Series Data
  15. Additional Resources

Feature Encoding: An Introduction

Feature encoding is a crucial step in data preprocessing for machine learning. It involves converting categorical variables into a numerical format that algorithms can understand and process. This transformation enables models to work with diverse data types and extract meaningful patterns.

import pandas as pd
from sklearn.preprocessing import LabelEncoder

# Sample data
data = {'Color': ['Red', 'Blue', 'Green', 'Red', 'Blue']}
df = pd.DataFrame(data)

# Label encoding
le = LabelEncoder()
df['Color_Encoded'] = le.fit_transform(df['Color'])

print(df)

One-Hot Encoding

One-hot encoding creates binary columns for each category in a feature. This method is useful when there's no ordinal relationship between categories.

import pandas as pd

# Sample data
data = {'Animal': ['Dog', 'Cat', 'Bird', 'Dog', 'Cat']}
df = pd.DataFrame(data)

# One-hot encoding
df_encoded = pd.get_dummies(df, columns=['Animal'], prefix='Is')

print(df_encoded)

Binary Encoding

Binary encoding represents each category as a binary number, then splits this number into separate columns. This method is memory-efficient for high-cardinality features.

from category_encoders import BinaryEncoder
import pandas as pd

# Sample data
data = {'Fruit': ['Apple', 'Banana', 'Cherry', 'Date', 'Apple']}
df = pd.DataFrame(data)

# Binary encoding
encoder = BinaryEncoder(columns=['Fruit'])
df_encoded = encoder.fit_transform(df)

print(df_encoded)

Ordinal Encoding

Ordinal encoding assigns an integer to each category based on its order. This method is suitable for features with a clear ranking.

from sklearn.preprocessing import OrdinalEncoder
import pandas as pd

# Sample data
data = {'Size': ['Small', 'Medium', 'Large', 'Medium', 'Small']}
df = pd.DataFrame(data)

# Ordinal encoding
encoder = OrdinalEncoder(categories=[['Small', 'Medium', 'Large']])
df['Size_Encoded'] = encoder.fit_transform(df[['Size']])

print(df)

Frequency Encoding

Frequency encoding replaces categories with their frequency in the dataset. This method can capture the relative importance of each category.

import pandas as pd

# Sample data
data = {'City': ['New York', 'London', 'Paris', 'New York', 'Tokyo', 'London']}
df = pd.DataFrame(data)

# Frequency encoding
frequency = df['City'].value_counts(normalize=True)
df['City_Encoded'] = df['City'].map(frequency)

print(df)

Target Encoding

Target encoding replaces a categorical value with the mean of the target variable for that category. This method can capture complex relationships between features and the target.

import pandas as pd

# Sample data
data = {'Product': ['A', 'B', 'C', 'A', 'B', 'C'],
        'Sales': [100, 200, 150, 120, 180, 160]}
df = pd.DataFrame(data)

# Target encoding
target_mean = df.groupby('Product')['Sales'].mean()
df['Product_Encoded'] = df['Product'].map(target_mean)

print(df)

Feature Hashing

Feature hashing uses a hash function to map high-dimensional categorical variables to a lower-dimensional space. This technique is memory-efficient and suitable for large datasets.

from sklearn.feature_extraction import FeatureHasher
import pandas as pd

# Sample data
data = {'Text': ['hello world', 'goodbye world', 'hello again']}
df = pd.DataFrame(data)

# Feature hashing
hasher = FeatureHasher(n_features=4, input_type='string')
hashed_features = hasher.transform(df['Text'])

print(hashed_features.toarray())

Embedding Encoding

Embedding encoding learns a dense vector representation for each category. This method is particularly useful for high-cardinality features and can capture complex relationships.

import tensorflow as tf
import numpy as np

# Sample data
vocab = ['apple', 'banana', 'cherry', 'date']
embedding_dim = 3

# Create embedding layer
embedding_layer = tf.keras.layers.Embedding(len(vocab), embedding_dim)

# Convert words to indices
word_indices = [vocab.index(word) for word in ['apple', 'banana', 'cherry']]

# Get embeddings
embeddings = embedding_layer(word_indices)

print(embeddings.numpy())

Cyclical Encoding

Cyclical encoding is useful for features with a circular nature, such as days of the week or hours in a day. It preserves the cyclic relationship between values.

import numpy as np
import pandas as pd

# Sample data
data = {'Hour': [0, 6, 12, 18, 23]}
df = pd.DataFrame(data)

# Cyclical encoding
df['Hour_Sin'] = np.sin(2 * np.pi * df['Hour'] / 24)
df['Hour_Cos'] = np.cos(2 * np.pi * df['Hour'] / 24)

print(df)

Real-Life Example: Weather Prediction

In weather prediction, we often deal with various categorical features. Let's encode some of these features for a machine learning model.

import pandas as pd
from sklearn.preprocessing import LabelEncoder

# Sample weather data
data = {
    'Day': ['Mon', 'Tue', 'Wed', 'Thu', 'Fri'],
    'WindDirection': ['N', 'NE', 'E', 'SE', 'S'],
    'Precipitation': ['None', 'Light', 'Moderate', 'Heavy', 'Light']
}
df = pd.DataFrame(data)

# Encoding
le = LabelEncoder()
df['Day_Encoded'] = le.fit_transform(df['Day'])
df['WindDirection_Encoded'] = le.fit_transform(df['WindDirection'])

# Ordinal encoding for Precipitation
precip_order = ['None', 'Light', 'Moderate', 'Heavy']
df['Precipitation_Encoded'] = df['Precipitation'].map(
    {val: idx for idx, val in enumerate(precip_order)}
)

print(df)

Real-Life Example: Text Classification

In text classification tasks, we often need to encode words or characters. Here's an example using feature hashing for sentiment analysis.

from sklearn.feature_extraction.text import HashingVectorizer
import pandas as pd

# Sample text data
data = {
    'Text': [
        "I love this product!",
        "This is terrible.",
        "Neutral opinion here.",
        "Absolutely amazing experience!"
    ],
    'Sentiment': ['Positive', 'Negative', 'Neutral', 'Positive']
}
df = pd.DataFrame(data)

# Feature hashing
vectorizer = HashingVectorizer(n_features=10)
hashed_features = vectorizer.transform(df['Text'])

# Add hashed features to dataframe
for i in range(10):
    df[f'Feature_{i}'] = hashed_features.getcol(i).toarray()

print(df)

Handling Missing Values in Categorical Features

When encoding categorical features, it's crucial to handle missing values appropriately. Here's an example using pandas to fill missing values before encoding.

import pandas as pd
from sklearn.preprocessing import LabelEncoder

# Sample data with missing values
data = {
    'Color': ['Red', 'Blue', None, 'Green', 'Blue', None],
    'Size': ['Small', None, 'Large', 'Medium', 'Small', 'Large']
}
df = pd.DataFrame(data)

# Fill missing values
df['Color'].fillna('Unknown', inplace=True)
df['Size'].fillna('Unknown', inplace=True)

# Label encoding
le = LabelEncoder()
df['Color_Encoded'] = le.fit_transform(df['Color'])
df['Size_Encoded'] = le.fit_transform(df['Size'])

print(df)

Combining Multiple Encoding Techniques

In real-world scenarios, you might need to apply different encoding techniques to various features. Here's an example combining multiple encoding methods.

import pandas as pd
from sklearn.preprocessing import LabelEncoder, OneHotEncoder
import numpy as np

# Sample data
data = {
    'Category': ['A', 'B', 'C', 'A', 'B'],
    'Subcategory': ['X', 'Y', 'Z', 'X', 'Z'],
    'Ordinal_Feature': ['Low', 'Medium', 'High', 'Low', 'High']
}
df = pd.DataFrame(data)

# Label Encoding for 'Category'
le = LabelEncoder()
df['Category_Encoded'] = le.fit_transform(df['Category'])

# One-Hot Encoding for 'Subcategory'
ohe = OneHotEncoder(sparse=False)
subcategory_encoded = ohe.fit_transform(df[['Subcategory']])
subcategory_columns = ohe.get_feature_names(['Subcategory'])
df[subcategory_columns] = subcategory_encoded

# Ordinal Encoding for 'Ordinal_Feature'
ordinal_map = {'Low': 0, 'Medium': 1, 'High': 2}
df['Ordinal_Encoded'] = df['Ordinal_Feature'].map(ordinal_map)

print(df)

Encoding for Time Series Data

Time series data often requires special encoding techniques to capture temporal patterns. Here's an example of encoding date-time features.

import pandas as pd
import numpy as np

# Generate sample time series data
date_rng = pd.date_range(start='2023-01-01', end='2023-12-31', freq='D')
df = pd.DataFrame(date_rng, columns=['Date'])
df['Value'] = np.random.randn(len(date_rng))

# Extract and encode time-based features
df['Year'] = df['Date'].dt.year
df['Month'] = df['Date'].dt.month
df['Day'] = df['Date'].dt.day
df['DayOfWeek'] = df['Date'].dt.dayofweek

# Cyclical encoding for month and day of week
df['Month_Sin'] = np.sin(2 * np.pi * df['Month'] / 12)
df['Month_Cos'] = np.cos(2 * np.pi * df['Month'] / 12)
df['DayOfWeek_Sin'] = np.sin(2 * np.pi * df['DayOfWeek'] / 7)
df['DayOfWeek_Cos'] = np.cos(2 * np.pi * df['DayOfWeek'] / 7)

print(df.head())

Additional Resources

For further exploration of feature encoding techniques and their applications in machine learning, consider the following resources:

  1. "A Comparative Study of Categorical Variable Encoding Techniques for Neural Networks" (arXiv:2003.07575) URL: https://arxiv.org/abs/2003.07575
  2. "An Empirical Study on Representation Learning for Spelling Error Correction" (arXiv:1909.08353) URL: https://arxiv.org/abs/1909.08353
  3. "Entity Embeddings of Categorical Variables" (arXiv:1604.06737) URL: https://arxiv.org/abs/1604.06737

These papers provide in-depth analyses of various encoding techniques and their effectiveness in different machine learning tasks.

More in Feature engineering and preprocessing

  1. Feature Engineering for Machine Learning using python4 min

    Feature engineering is the process of transforming raw data into features that better represent the underlying problem to the predictive models, resulting in...

  2. Diving into Train, Validation, and Test Data in Machine Learning with Python7 min

    In machine learning, the division of data into train, validation, and test sets is crucial for developing robust models. This process helps in training the...

  3. Feature Engineering Techniques for Data Science8 min

    Feature engineering is the process of transforming raw data into meaningful features that enhance machine learning model performance. It involves selecting...

  4. Diving Deeper into Feature Scaling in Machine Learning with Python9 min

    Feature scaling is a crucial preprocessing step in machine learning that transforms the features of a dataset to a common scale. This process ensures that...

  5. Feature Importance in Machine Learning with Python9 min

    Feature importance is a crucial concept in machine learning that helps identify which features in a dataset have the most significant impact on the model's...

  6. De-normalized Data for Machine Learning Models in Python8 min

    De-normalized data refers to a data structure where redundant information is intentionally added to improve query performance or simplify data handling. This...