xbe.at

Machine learning / Fundamentals and overview

Beginner's Guide to Machine Learning with Python

12 min read, 14 sections

Contents
  1. Python Data Structures for Machine Learning
  2. Data Preprocessing Pipeline
  3. Linear Regression Implementation
  4. Mathematical Foundations of Gradient Descent
  5. K-Means Clustering Implementation
  6. Neural Network Architecture
  7. Decision Tree Implementation
  8. Support Vector Machine Core Algorithm
  9. Ensemble Methods Implementation
  10. Deep Learning Layer Implementation
  11. Convolutional Neural Network Core
  12. Natural Language Processing Basics
  13. Reinforcement Learning Implementation
  14. Additional Resources

Python Data Structures for Machine Learning

Understanding fundamental data structures is crucial for efficient machine learning implementation. NumPy arrays and Pandas DataFrames form the backbone of data manipulation, offering optimized operations for numerical computations and data analysis in machine learning applications.

import numpy as np
import pandas as pd

# Creating a feature matrix using NumPy
X = np.array([[1, 2, 3],
              [4, 5, 6],
              [7, 8, 9]])

# Creating a DataFrame with labeled data
df = pd.DataFrame(X, columns=['feature1', 'feature2', 'feature3'])
print("NumPy Array Shape:", X.shape)
print("\nDataFrame Head:\n", df.head())

# Basic statistical operations
print("\nFeature Statistics:\n", df.describe())

Data Preprocessing Pipeline

Data preprocessing is a critical step that significantly impacts model performance. This implementation demonstrates a comprehensive pipeline including handling missing values, feature scaling, and categorical encoding using scikit-learn.

from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.impute import SimpleImputer

def preprocess_pipeline(X, categorical_cols=None):
    # Handle missing values
    imputer = SimpleImputer(strategy='mean')
    X_imputed = imputer.fit_transform(X)
    
    # Scale numerical features
    scaler = StandardScaler()
    X_scaled = scaler.fit_transform(X_imputed)
    
    # Encode categorical variables if present
    if categorical_cols:
        le = LabelEncoder()
        for col in categorical_cols:
            X[:, col] = le.fit_transform(X[:, col])
    
    return X_scaled

# Example usage
X = np.array([[1, np.nan, 3], [4, 5, 6], [7, 8, np.nan]])
X_processed = preprocess_pipeline(X)
print("Processed Data:\n", X_processed)

Linear Regression Implementation

Linear regression serves as the foundation for understanding supervised learning. This implementation builds a linear regression model from scratch, including gradient descent optimization and mean squared error calculation.

class LinearRegression:
    def __init__(self, learning_rate=0.01, iterations=1000):
        self.lr = learning_rate
        self.iterations = iterations
        self.weights = None
        self.bias = None
    
    def fit(self, X, y):
        n_samples, n_features = X.shape
        self.weights = np.zeros(n_features)
        self.bias = 0
        
        for _ in range(self.iterations):
            y_pred = np.dot(X, self.weights) + self.bias
            
            # Gradient descent
            dw = (1/n_samples) * np.dot(X.T, (y_pred - y))
            db = (1/n_samples) * np.sum(y_pred - y)
            
            self.weights -= self.lr * dw
            self.bias -= self.lr * db
    
    def predict(self, X):
        return np.dot(X, self.weights) + self.bias

# Example usage
X = np.array([[1], [2], [3], [4]])
y = np.array([2, 4, 6, 8])
model = LinearRegression()
model.fit(X, y)
print("Predictions:", model.predict(np.array([[5], [6]])))

Mathematical Foundations of Gradient Descent

Understanding the mathematical principles behind gradient descent is essential for optimizing machine learning models. The process involves calculating partial derivatives and updating parameters iteratively.

# Mathematical representation of gradient descent
"""
Cost Function (Mean Squared Error):
$$J(\theta) = \frac{1}{2m} \sum_{i=1}^{m} (h_\theta(x^{(i)}) - y^{(i)})^2$$

Gradient Descent Update Rule:
$$\theta_j := \theta_j - \alpha \frac{\partial}{\partial \theta_j} J(\theta)$$

where:
$$\frac{\partial}{\partial \theta_j} J(\theta) = \frac{1}{m} \sum_{i=1}^{m} (h_\theta(x^{(i)}) - y^{(i)}) x_j^{(i)}$$
"""

def gradient_descent_step(X, y, theta, alpha):
    m = len(y)
    h = np.dot(X, theta)
    gradient = (1/m) * np.dot(X.T, (h - y))
    theta -= alpha * gradient
    return theta

K-Means Clustering Implementation

K-means clustering is a fundamental unsupervised learning algorithm. This implementation demonstrates cluster centroid initialization, distance calculation, and iterative optimization for finding optimal clusters.

class KMeans:
    def __init__(self, n_clusters=3, max_iters=100):
        self.n_clusters = n_clusters
        self.max_iters = max_iters
        self.centroids = None
    
    def fit(self, X):
        # Random initialization of centroids
        idx = np.random.choice(len(X), self.n_clusters, replace=False)
        self.centroids = X[idx]
        
        for _ in range(self.max_iters):
            # Assign points to nearest centroid
            distances = np.sqrt(((X - self.centroids[:, np.newaxis])**2).sum(axis=2))
            labels = np.argmin(distances, axis=0)
            
            # Update centroids
            new_centroids = np.array([X[labels == k].mean(axis=0)
                                    for k in range(self.n_clusters)])
            
            if np.all(self.centroids == new_centroids):
                break
                
            self.centroids = new_centroids
        
        return labels

# Example usage
X = np.random.randn(100, 2)
kmeans = KMeans(n_clusters=3)
labels = kmeans.fit(X)
print("Cluster Assignments:", labels[:10])

Neural Network Architecture

A neural network implementation showcasing the fundamental components of deep learning, including forward propagation, activation functions, and backpropagation using numpy for matrix operations and gradient calculations.

class NeuralNetwork:
    def __init__(self, layers=[3, 4, 1], learning_rate=0.1):
        self.layers = layers
        self.lr = learning_rate
        self.weights = [np.random.randn(layers[i], layers[i+1]) / np.sqrt(layers[i]) 
                       for i in range(len(layers)-1)]
        self.biases = [np.zeros((1, layers[i+1])) for i in range(len(layers)-1)]
    
    def sigmoid(self, x):
        return 1 / (1 + np.exp(-x))
    
    def sigmoid_derivative(self, x):
        return x * (1 - x)
    
    def forward(self, X):
        self.activations = [X]
        for i in range(len(self.weights)):
            net = np.dot(self.activations[-1], self.weights[i]) + self.biases[i]
            self.activations.append(self.sigmoid(net))
        return self.activations[-1]
    
    def backward(self, X, y):
        m = X.shape[0]
        delta = (self.activations[-1] - y) * self.sigmoid_derivative(self.activations[-1])
        
        for i in range(len(self.weights) - 1, -1, -1):
            self.weights[i] -= self.lr * np.dot(self.activations[i].T, delta) / m
            self.biases[i] -= self.lr * np.sum(delta, axis=0, keepdims=True) / m
            if i > 0:
                delta = np.dot(delta, self.weights[i].T) * self.sigmoid_derivative(self.activations[i])

# Example usage
nn = NeuralNetwork([2, 4, 1])
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.array([[0], [1], [1], [0]])
predictions = nn.forward(X)
print("Initial predictions:", predictions)

Decision Tree Implementation

Decision trees form the basis for many ensemble methods. This implementation shows how to build a decision tree from scratch, including information gain calculation and recursive tree construction.

class DecisionTreeNode:
    def __init__(self, feature=None, threshold=None, left=None, right=None, value=None):
        self.feature = feature
        self.threshold = threshold
        self.left = left
        self.right = right
        self.value = value

class DecisionTree:
    def __init__(self, max_depth=5):
        self.max_depth = max_depth
        self.root = None
    
    def entropy(self, y):
        classes = np.unique(y)
        entropy = 0
        for cls in classes:
            p = len(y[y == cls]) / len(y)
            entropy -= p * np.log2(p)
        return entropy
    
    def information_gain(self, X, y, feature, threshold):
        parent_entropy = self.entropy(y)
        
        left_mask = X[:, feature] <= threshold
        right_mask = ~left_mask
        
        if np.sum(left_mask) == 0 or np.sum(right_mask) == 0:
            return 0
        
        left_entropy = self.entropy(y[left_mask])
        right_entropy = self.entropy(y[right_mask])
        
        left_weight = np.sum(left_mask) / len(y)
        right_weight = np.sum(right_mask) / len(y)
        
        information_gain = parent_entropy - (left_weight * left_entropy + 
                                          right_weight * right_entropy)
        return information_gain
    
    def build_tree(self, X, y, depth=0):
        n_samples, n_features = X.shape
        n_classes = len(np.unique(y))
        
        if depth >= self.max_depth or n_classes == 1:
            return DecisionTreeNode(value=np.argmax(np.bincount(y)))
        
        best_gain = -1
        best_split = None
        
        for feature in range(n_features):
            thresholds = np.unique(X[:, feature])
            for threshold in thresholds:
                gain = self.information_gain(X, y, feature, threshold)
                if gain > best_gain:
                    best_gain = gain
                    best_split = (feature, threshold)
        
        if best_split is None:
            return DecisionTreeNode(value=np.argmax(np.bincount(y)))
        
        feature, threshold = best_split
        left_mask = X[:, feature] <= threshold
        right_mask = ~left_mask
        
        left_subtree = self.build_tree(X[left_mask], y[left_mask], depth + 1)
        right_subtree = self.build_tree(X[right_mask], y[right_mask], depth + 1)
        
        return DecisionTreeNode(feature, threshold, left_subtree, right_subtree)

# Example usage
X = np.array([[1, 2], [3, 4], [5, 6], [7, 8]])
y = np.array([0, 0, 1, 1])
tree = DecisionTree(max_depth=3)
tree.root = tree.build_tree(X, y)

Support Vector Machine Core Algorithm

Support Vector Machines optimize the margin between classes using quadratic programming. This implementation demonstrates the key concepts of kernel tricks and margin maximization.

class SVM:
    def __init__(self, learning_rate=0.001, lambda_param=0.01, n_iterations=1000):
        self.lr = learning_rate
        self.lambda_param = lambda_param
        self.n_iterations = n_iterations
        self.w = None
        self.b = None
    
    def gaussian_kernel(self, x1, x2, sigma=1.0):
        return np.exp(-np.linalg.norm(x1-x2)**2 / (2*(sigma**2)))
    
    def fit(self, X, y):
        n_samples, n_features = X.shape
        y_ = np.where(y <= 0, -1, 1)
        
        self.w = np.zeros(n_features)
        self.b = 0
        
        for _ in range(self.n_iterations):
            for idx, x_i in enumerate(X):
                condition = y_[idx] * (np.dot(x_i, self.w) + self.b) >= 1
                if condition:
                    self.w -= self.lr * (2 * self.lambda_param * self.w)
                else:
                    self.w -= self.lr * (2 * self.lambda_param * self.w - 
                                       np.dot(x_i, y_[idx]))
                    self.b -= self.lr * y_[idx]
    
    def predict(self, X):
        linear_output = np.dot(X, self.w) + self.b
        return np.sign(linear_output)

# Example usage
X = np.array([[1, 2], [2, 3], [4, 5], [5, 6]])
y = np.array([-1, -1, 1, 1])
svm = SVM()
svm.fit(X, y)
predictions = svm.predict(np.array([[3, 4]]))
print("SVM Prediction:", predictions)

Ensemble Methods Implementation

Ensemble methods combine multiple models to create more robust predictions. This implementation showcases Random Forest and Gradient Boosting techniques using base decision trees as weak learners.

class RandomForest:
    def __init__(self, n_trees=10, max_depth=5):
        self.n_trees = n_trees
        self.max_depth = max_depth
        self.trees = []
    
    def bootstrap_sample(self, X, y):
        n_samples = X.shape[0]
        idxs = np.random.choice(n_samples, n_samples, replace=True)
        return X[idxs], y[idxs]
    
    def fit(self, X, y):
        self.trees = []
        for _ in range(self.n_trees):
            tree = DecisionTree(max_depth=self.max_depth)
            X_sample, y_sample = self.bootstrap_sample(X, y)
            tree.root = tree.build_tree(X_sample, y_sample)
            self.trees.append(tree)
    
    def predict(self, X):
        tree_preds = np.array([tree.predict(X) for tree in self.trees])
        return np.round(np.mean(tree_preds, axis=0))

# Example usage
X = np.random.randn(100, 4)
y = (X[:, 0] + X[:, 1] > 0).astype(int)
rf = RandomForest(n_trees=10, max_depth=3)
rf.fit(X, y)
predictions = rf.predict(X[:5])
print("Random Forest Predictions:", predictions)

Deep Learning Layer Implementation

Understanding the internal mechanics of neural network layers is crucial. This implementation shows custom layer implementations including dense, dropout, and batch normalization layers.

class Layer:
    def __init__(self):
        self.input = None
        self.output = None
    
    def forward(self, input):
        pass
    
    def backward(self, output_gradient, learning_rate):
        pass

class Dense(Layer):
    def __init__(self, input_size, output_size):
        super().__init__()
        self.weights = np.random.randn(input_size, output_size) * 0.01
        self.bias = np.zeros((1, output_size))
    
    def forward(self, input):
        self.input = input
        return np.dot(input, self.weights) + self.bias
    
    def backward(self, output_gradient, learning_rate):
        weights_gradient = np.dot(self.input.T, output_gradient)
        input_gradient = np.dot(output_gradient, self.weights.T)
        
        self.weights -= learning_rate * weights_gradient
        self.bias -= learning_rate * output_gradient.mean(axis=0, keepdims=True)
        
        return input_gradient

class Dropout(Layer):
    def __init__(self, rate):
        super().__init__()
        self.rate = rate
        self.mask = None
    
    def forward(self, input, training=True):
        if not training:
            return input
        
        self.mask = np.random.binomial(1, 1-self.rate, input.shape) / (1-self.rate)
        return input * self.mask
    
    def backward(self, output_gradient, learning_rate):
        return output_gradient * self.mask

# Example usage
layer = Dense(4, 3)
X = np.random.randn(5, 4)
output = layer.forward(X)
print("Dense Layer Output Shape:", output.shape)

Convolutional Neural Network Core

Implementation of core CNN operations including convolution, pooling, and flattening layers for image processing tasks.

class Convolution2D(Layer):
    def __init__(self, input_shape, kernel_size, depth):
        self.input_shape = input_shape
        self.input_depth = input_shape[2]
        self.kernel_size = kernel_size
        self.depth = depth
        self.output_shape = (input_shape[0] - kernel_size + 1,
                           input_shape[1] - kernel_size + 1,
                           depth)
        
        self.kernels = np.random.randn(kernel_size, kernel_size, 
                                     self.input_depth, depth) * 0.1
        self.biases = np.zeros(depth)
    
    def forward(self, input):
        self.input = input
        self.output = np.zeros(self.output_shape)
        
        for i in range(self.output_shape[0]):
            for j in range(self.output_shape[1]):
                input_slice = input[i:i+self.kernel_size, 
                                  j:j+self.kernel_size]
                for d in range(self.depth):
                    self.output[i, j, d] = np.sum(input_slice * 
                                                 self.kernels[:,:,:,d]) + self.biases[d]
        return self.output

class MaxPool2D(Layer):
    def __init__(self, pool_size=2, stride=2):
        self.pool_size = pool_size
        self.stride = stride
    
    def forward(self, input):
        self.input = input
        h, w, d = input.shape
        h_out = (h - self.pool_size) // self.stride + 1
        w_out = (w - self.pool_size) // self.stride + 1
        
        output = np.zeros((h_out, w_out, d))
        
        for i in range(h_out):
            for j in range(w_out):
                h_start = i * self.stride
                h_end = h_start + self.pool_size
                w_start = j * self.stride
                w_end = w_start + self.pool_size
                
                pool_slice = input[h_start:h_end, w_start:w_end, :]
                output[i, j, :] = np.max(pool_slice, axis=(0,1))
        
        return output

# Example usage
input_data = np.random.randn(28, 28, 1)  # Sample image
conv_layer = Convolution2D((28, 28, 1), 3, 32)
pool_layer = MaxPool2D()

conv_output = conv_layer.forward(input_data)
pool_output = pool_layer.forward(conv_output)
print("Conv Output Shape:", conv_output.shape)
print("Pool Output Shape:", pool_output.shape)

Natural Language Processing Basics

Implementation of fundamental NLP preprocessing techniques and word embeddings, showcasing text vectorization and basic sequence processing for machine learning applications.

class TextProcessor:
    def __init__(self, max_vocab_size=10000):
        self.max_vocab_size = max_vocab_size
        self.word2idx = {}
        self.idx2word = {}
        self.word_freq = {}
        self.vocab_size = 0
    
    def build_vocab(self, texts):
        # Count word frequencies
        for text in texts:
            for word in text.lower().split():
                self.word_freq[word] = self.word_freq.get(word, 0) + 1
        
        # Sort by frequency and limit vocabulary size
        sorted_words = sorted(self.word_freq.items(), 
                            key=lambda x: x[1], reverse=True)
        vocab_words = sorted_words[:self.max_vocab_size]
        
        # Build word-to-index mappings
        for idx, (word, _) in enumerate(vocab_words):
            self.word2idx[word] = idx
            self.idx2word[idx] = word
        
        self.vocab_size = len(self.word2idx)
    
    def tokenize(self, text):
        return [self.word2idx.get(word.lower(), self.word2idx['<UNK>']) 
                for word in text.split()]
    
    def create_embeddings(self, embedding_dim=100):
        return np.random.randn(self.vocab_size, embedding_dim) * 0.1

# Example usage
texts = [
    "machine learning is fascinating",
    "deep learning revolutionizes AI",
    "neural networks are powerful"
]

processor = TextProcessor(max_vocab_size=100)
processor.build_vocab(texts)
embeddings = processor.create_embeddings(embedding_dim=50)

sample_text = "machine learning is powerful"
tokens = processor.tokenize(sample_text)
print("Tokenized sequence:", tokens)
print("Embedding matrix shape:", embeddings.shape)

Reinforcement Learning Implementation

A basic Q-learning implementation demonstrating fundamental concepts of reinforcement learning including state-action values, exploration-exploitation, and policy updates.

class QLearningAgent:
    def __init__(self, n_states, n_actions, learning_rate=0.1, 
                 discount_factor=0.95, epsilon=0.1):
        self.q_table = np.zeros((n_states, n_actions))
        self.lr = learning_rate
        self.gamma = discount_factor
        self.epsilon = epsilon
        self.n_actions = n_actions
    
    def choose_action(self, state):
        if np.random.random() < self.epsilon:
            return np.random.randint(self.n_actions)
        return np.argmax(self.q_table[state])
    
    def learn(self, state, action, reward, next_state):
        old_value = self.q_table[state, action]
        next_max = np.max(self.q_table[next_state])
        
        # Q-learning update rule
        new_value = (1 - self.lr) * old_value + \
                   self.lr * (reward + self.gamma * next_max)
        
        self.q_table[state, action] = new_value
    
    def decay_epsilon(self, decay_rate=0.995):
        self.epsilon *= decay_rate

class Environment:
    def __init__(self, n_states=10):
        self.n_states = n_states
        self.current_state = 0
    
    def step(self, action):
        # Simplified environment dynamics
        if action == 1:  # Move right
            self.current_state = min(self.current_state + 1, self.n_states - 1)
        else:  # Move left
            self.current_state = max(self.current_state - 1, 0)
        
        # Reward structure
        reward = 1 if self.current_state == self.n_states - 1 else 0
        done = self.current_state == self.n_states - 1
        
        return self.current_state, reward, done

# Example usage
env = Environment(n_states=10)
agent = QLearningAgent(n_states=10, n_actions=2)

# Training loop
for episode in range(100):
    state = 0
    done = False
    total_reward = 0
    
    while not done:
        action = agent.choose_action(state)
        next_state, reward, done = env.step(action)
        agent.learn(state, action, reward, next_state)
        state = next_state
        total_reward += reward
    
    agent.decay_epsilon()
    if episode % 10 == 0:
        print(f"Episode {episode}, Total Reward: {total_reward}")

Additional Resources

More in Fundamentals and overview

  1. Building a Machine Learning Model11 min

    Advanced data preprocessing techniques including handling missing values, feature scaling, and categorical encoding using pandas and scikit-learn libraries...

  2. AI vs Machine Learning Exploring the Differences11 min

    Artificial Intelligence (AI) and Machine Learning (ML) are often used interchangeably, but they have distinct differences. AI is a broader concept that aims...

  3. Building a Machine Learning Model The Complete Workflow in Python4 min

    Machine learning is a powerful tool for solving complex problems. This slideshow will guide you through the complete workflow of building a machine learning...

  4. Advanced Machine Learning Models in Python11 min

    Advanced Machine Learning Models are sophisticated algorithms that can learn complex patterns from data. These models go beyond traditional methods, offering...

  5. Comparing CSV Files Using Machine Learning in Python7 min

    Machine learning techniques can be applied to compare CSV files for similarity. This process involves data preprocessing, feature extraction, and utilizing...

  6. Acing the Machine Learning Interview11 min

    Dropout is a powerful regularization technique used in deep learning to combat overfitting. It works by randomly "turning off" a certain percentage of...