xbe.at

Machine learning / Fundamentals and overview

Beginner's Guide to Python for Data Science and Machine Learning

8 min read, 15 sections

Contents
  1. Loading and Examining Data with Pandas
  2. Data Cleaning and Preprocessing
  3. Feature Engineering
  4. Data Aggregation and Grouping
  5. Time Series Analysis with Pandas
  6. Advanced Data Visualization
  7. Linear Regression Implementation
  8. Time Series Forecasting with ARIMA
  9. Feature Selection Using Statistical Methods
  10. Cross-Validation and Model Evaluation
  11. Principal Component Analysis (PCA)
  12. Clustering Analysis with K-means
  13. Advanced Text Processing with NLTK
  14. Time Series Anomaly Detection
  15. Additional Resources

Loading and Examining Data with Pandas

Pandas is a powerful data manipulation library that provides essential tools for data analysis. The first step in any data science project is loading and examining your dataset to understand its structure, size, and basic characteristics.

import pandas as pd
import numpy as np

# Load dataset from CSV file
df = pd.read_csv('data.csv')

# Display basic information about the dataset
print("Dataset Info:")
print(df.info())

# Display first 5 rows and basic statistics
print("\nFirst 5 rows:")
print(df.head())
print("\nBasic Statistics:")
print(df.describe())

# Check for missing values
print("\nMissing Values:")
print(df.isnull().sum())

Data Cleaning and Preprocessing

Data cleaning is crucial for ensuring accurate analysis. This process involves handling missing values, removing duplicates, and dealing with outliers to prepare your dataset for further analysis.

# Handle missing values
df.fillna(df.mean(), inplace=True)  # Fill numeric columns with mean
df['categorical_col'].fillna(df['categorical_col'].mode()[0], inplace=True)

# Remove duplicates
df.drop_duplicates(inplace=True)

# Handle outliers using IQR method
def remove_outliers(df, column):
    Q1 = df[column].quantile(0.25)
    Q3 = df[column].quantile(0.75)
    IQR = Q3 - Q1
    lower_bound = Q1 - 1.5 * IQR
    upper_bound = Q3 + 1.5 * IQR
    return df[(df[column] >= lower_bound) & (df[column] <= upper_bound)]

# Apply to numeric columns
numeric_cols = df.select_dtypes(include=[np.number]).columns
for col in numeric_cols:
    df = remove_outliers(df, col)

Feature Engineering

Feature engineering transforms raw data into meaningful features that better represent the underlying patterns in your data, improving model performance and providing deeper insights.

# Create new features
df['year'] = pd.to_datetime(df['date']).dt.year
df['month'] = pd.to_datetime(df['date']).dt.month
df['day_of_week'] = pd.to_datetime(df['date']).dt.dayofweek

# Binning continuous variables
df['age_group'] = pd.cut(df['age'], 
                        bins=[0, 18, 35, 50, 65, 100],
                        labels=['0-18', '19-35', '36-50', '51-65', '65+'])

# One-hot encoding for categorical variables
df = pd.get_dummies(df, columns=['category'], prefix='cat')

# Feature scaling
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
df[numeric_cols] = scaler.fit_transform(df[numeric_cols])

Data Aggregation and Grouping

Understanding how to aggregate and group data is essential for extracting meaningful insights and patterns from your dataset, allowing you to analyze trends across different categories.

# Basic groupby operations
group_stats = df.groupby('category').agg({
    'sales': ['mean', 'sum', 'count'],
    'profit': ['mean', 'sum'],
    'quantity': 'sum'
}).round(2)

# Multiple level grouping
multi_group = df.groupby(['region', 'category']).agg({
    'sales': 'sum',
    'profit': 'mean'
}).reset_index()

# Pivot tables
pivot_table = pd.pivot_table(df,
                           values='sales',
                           index='region',
                           columns='category',
                           aggfunc='sum',
                           fill_value=0)

Time Series Analysis with Pandas

Time series analysis is fundamental in data science for analyzing temporal patterns. Pandas provides powerful functionality for handling datetime data, resampling, and calculating rolling statistics.

# Convert to datetime and set as index
df['date'] = pd.to_datetime(df['date'])
df.set_index('date', inplace=True)

# Resample data to monthly frequency
monthly_data = df['sales'].resample('M').sum()

# Calculate rolling statistics
rolling_mean = df['sales'].rolling(window=7).mean()
rolling_std = df['sales'].rolling(window=7).std()

# Calculate year-over-year growth
yoy_growth = df['sales'].pct_change(periods=12) * 100

# Seasonal decomposition
from statsmodels.tsa.seasonal import seasonal_decompose
decomposition = seasonal_decompose(df['sales'], period=12)

Advanced Data Visualization

Effective data visualization is crucial for understanding patterns and communicating insights. This implementation combines pandas with popular visualization libraries for comprehensive data exploration.

import matplotlib.pyplot as plt
import seaborn as sns

# Set style
plt.style.use('seaborn')
sns.set_palette("husl")

# Create subplots
fig, axes = plt.subplots(2, 2, figsize=(15, 12))

# Distribution plot
sns.histplot(data=df, x='sales', kde=True, ax=axes[0,0])
axes[0,0].set_title('Sales Distribution')

# Box plot
sns.boxplot(data=df, x='category', y='sales', ax=axes[0,1])
axes[0,1].set_title('Sales by Category')

# Time series plot
df['sales'].plot(ax=axes[1,0])
axes[1,0].set_title('Sales Over Time')

# Correlation heatmap
sns.heatmap(df.corr(), annot=True, cmap='coolwarm', ax=axes[1,1])
axes[1,1].set_title('Correlation Matrix')

plt.tight_layout()
plt.show()

Linear Regression Implementation

Linear regression serves as a fundamental building block in machine learning. This implementation shows how to prepare data, train a model, and evaluate its performance using scikit-learn.

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score

# Prepare features and target
X = df[['feature1', 'feature2', 'feature3']]
y = df['target']

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train model
model = LinearRegression()
model.fit(X_train, y_train)

# Make predictions
y_pred = model.predict(X_test)

# Evaluate model
print(f"R² Score: {r2_score(y_test, y_pred):.4f}")
print(f"RMSE: {np.sqrt(mean_squared_error(y_test, y_pred)):.4f}")

# Feature importance
for feature, coef in zip(X.columns, model.coef_):
    print(f"{feature}: {coef:.4f}")

Time Series Forecasting with ARIMA

The ARIMA model is widely used for time series forecasting. This implementation demonstrates how to analyze, model, and forecast time series data using statsmodels.

from statsmodels.tsa.arima.model import ARIMA
from statsmodels.tsa.stattools import adfuller

# Check stationarity
def check_stationarity(timeseries):
    result = adfuller(timeseries)
    print('ADF Statistic:', result[0])
    print('p-value:', result[1])

# Fit ARIMA model
model = ARIMA(df['sales'], order=(1,1,1))
results = model.fit()

# Make forecast
forecast = results.forecast(steps=30)

# Plot results
plt.figure(figsize=(12, 6))
plt.plot(df.index, df['sales'], label='Observed')
plt.plot(forecast.index, forecast, label='Forecast')
plt.title('ARIMA Forecast')
plt.legend()
plt.show()

# Print model summary
print(results.summary())

Feature Selection Using Statistical Methods

Feature selection is crucial for building efficient and accurate models. This implementation demonstrates various statistical methods to identify the most relevant features in your dataset.

from sklearn.feature_selection import SelectKBest, f_classif, mutual_info_classif
from sklearn.ensemble import RandomForestClassifier

# Univariate feature selection
selector = SelectKBest(score_func=f_classif, k=5)
X_selected = selector.fit_transform(X, y)

# Get feature scores
scores = pd.DataFrame({
    'Feature': X.columns,
    'Score': selector.scores_
}).sort_values('Score', ascending=False)

# Random Forest feature importance
rf = RandomForestClassifier(n_estimators=100)
rf.fit(X, y)

# Calculate feature importance
importances = pd.DataFrame({
    'Feature': X.columns,
    'Importance': rf.feature_importances_
}).sort_values('Importance', ascending=False)

print("Top 5 Features by F-score:")
print(scores.head())
print("\nTop 5 Features by Random Forest:")
print(importances.head())

Cross-Validation and Model Evaluation

Proper model evaluation is essential for ensuring reliable performance estimates. This implementation shows various cross-validation techniques and evaluation metrics.

from sklearn.model_selection import cross_val_score, KFold
from sklearn.metrics import confusion_matrix, classification_report
from sklearn.ensemble import RandomForestClassifier

# Initialize model
model = RandomForestClassifier(n_estimators=100, random_state=42)

# Perform k-fold cross-validation
kfold = KFold(n_splits=5, shuffle=True, random_state=42)
cv_scores = cross_val_score(model, X, y, cv=kfold, scoring='accuracy')

# Train final model and evaluate
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

# Print evaluation metrics
print(f"Cross-validation scores: {cv_scores}")
print(f"Mean CV Score: {cv_scores.mean():.4f} (+/- {cv_scores.std()*2:.4f})")
print("\nConfusion Matrix:")
print(confusion_matrix(y_test, y_pred))
print("\nClassification Report:")
print(classification_report(y_test, y_pred))

Principal Component Analysis (PCA)

PCA is a powerful dimensionality reduction technique that helps visualize high-dimensional data and reduce feature space while preserving important information.

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

# Standardize features
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Apply PCA
pca = PCA()
X_pca = pca.fit_transform(X_scaled)

# Calculate explained variance ratio
explained_variance = pd.DataFrame(
    pca.explained_variance_ratio_,
    columns=['Explained Variance Ratio'],
    index=[f'PC{i+1}' for i in range(len(pca.components_))]
)

# Plot cumulative variance explained
plt.figure(figsize=(10, 6))
plt.plot(np.cumsum(pca.explained_variance_ratio_))
plt.xlabel('Number of Components')
plt.ylabel('Cumulative Explained Variance Ratio')
plt.title('PCA Analysis - Explained Variance')
plt.grid(True)
plt.show()

Clustering Analysis with K-means

Clustering helps identify natural groupings in your data. This implementation shows how to perform and evaluate k-means clustering with visualization.

from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score

# Find optimal number of clusters using elbow method
inertias = []
silhouette_scores = []
K = range(2, 11)

for k in K:
    kmeans = KMeans(n_clusters=k, random_state=42)
    kmeans.fit(X_scaled)
    inertias.append(kmeans.inertia_)
    silhouette_scores.append(silhouette_score(X_scaled, kmeans.labels_))

# Plot elbow curve
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 5))

ax1.plot(K, inertias, 'bx-')
ax1.set_xlabel('k')
ax1.set_ylabel('Inertia')
ax1.set_title('Elbow Method')

ax2.plot(K, silhouette_scores, 'rx-')
ax2.set_xlabel('k')
ax2.set_ylabel('Silhouette Score')
ax2.set_title('Silhouette Analysis')

plt.show()

# Fit final model with optimal k
optimal_k = 3  # Based on analysis
kmeans = KMeans(n_clusters=optimal_k, random_state=42)
clusters = kmeans.fit_predict(X_scaled)

Advanced Text Processing with NLTK

Natural Language Processing is essential for analyzing textual data. This implementation demonstrates key text processing techniques using NLTK for data science applications.

import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from nltk.stem import WordNetLemmatizer
from collections import Counter

# Download required NLTK data
nltk.download(['punkt', 'stopwords', 'wordnet'])

def preprocess_text(text):
    # Tokenization
    tokens = word_tokenize(text.lower())
    
    # Remove stopwords and non-alphabetic tokens
    stop_words = set(stopwords.words('english'))
    tokens = [token for token in tokens if token.isalpha() and token not in stop_words]
    
    # Lemmatization
    lemmatizer = WordNetLemmatizer()
    tokens = [lemmatizer.lemmatize(token) for token in tokens]
    
    return tokens

# Example usage
text_data = df['text_column'].tolist()
processed_texts = [preprocess_text(text) for text in text_data]

# Get word frequencies
all_words = [word for text in processed_texts for word in text]
word_freq = Counter(all_words)

# Print most common words
print("Most common words:")
print(word_freq.most_common(10))

Time Series Anomaly Detection

Anomaly detection in time series data is crucial for identifying unusual patterns. This implementation shows how to detect anomalies using statistical methods and visualization.

import numpy as np
from scipy import stats

def detect_anomalies(series, window=12, sigma=3):
    # Calculate rolling statistics
    rolling_mean = series.rolling(window=window).mean()
    rolling_std = series.rolling(window=window).std()
    
    # Calculate z-scores
    z_scores = (series - rolling_mean) / rolling_std
    
    # Identify anomalies
    anomalies = np.abs(z_scores) > sigma
    
    return anomalies, z_scores

# Apply to time series data
anomalies, z_scores = detect_anomalies(df['value'])

# Visualize results
plt.figure(figsize=(15, 8))
plt.subplot(211)
plt.plot(df.index, df['value'], label='Original')
plt.plot(df.index[anomalies], df['value'][anomalies], 'ro', label='Anomalies')
plt.title('Time Series with Anomalies')
plt.legend()

plt.subplot(212)
plt.plot(df.index, z_scores, label='Z-scores')
plt.axhline(y=3, color='r', linestyle='--', label='Upper Threshold')
plt.axhline(y=-3, color='r', linestyle='--', label='Lower Threshold')
plt.title('Z-scores with Thresholds')
plt.legend()
plt.tight_layout()
plt.show()

Additional Resources

More in Fundamentals and overview

  1. Building a Machine Learning Model The Complete Workflow in Python4 min

    Machine learning is a powerful tool for solving complex problems. This slideshow will guide you through the complete workflow of building a machine learning...

  2. Balancing Accuracy and Interpretability in Machine Learning with Python11 min

    In machine learning, achieving high accuracy is often the primary goal. However, interpretability is equally important, especially in critical domains like...

  3. Comparing CSV Files Using Machine Learning in Python7 min

    Machine learning techniques can be applied to compare CSV files for similarity. This process involves data preprocessing, feature extraction, and utilizing...

  4. AI vs Machine Learning Exploring the Differences11 min

    Artificial Intelligence (AI) and Machine Learning (ML) are often used interchangeably, but they have distinct differences. AI is a broader concept that aims...

  5. Comparing Gini Impurity and Entropy in Decision Trees14 min

    The Gini impurity metric quantifies the probability of incorrect classification when randomly choosing an element from a dataset based on the class...

  6. Advanced Machine Learning Models in Python11 min

    Advanced Machine Learning Models are sophisticated algorithms that can learn complex patterns from data. These models go beyond traditional methods, offering...