xbe.at

Miscellany / AI applications and systems

Kernel Activation Networks Flexible Neural Architectures:

12 min read, 14 sections

Contents
  1. Introduction to Kernel Activation Networks (KANs)
  2. Traditional Neural Networks
  3. Kernel Activation Networks (KANs)
  4. Comparison of Traditional NNs and KANs
  5. Advantages of KANs
  6. Challenges and Considerations
  7. Implementation Considerations
  8. Real-life Example: Image Classification
  9. Real-life Example: Natural Language Processing
  10. KAN Architecture Variations
  11. Training Strategies for KANs
  12. Interpreting KAN Behaviors
  13. Future Directions and Research Opportunities
  14. Additional Resources

Introduction to Kernel Activation Networks (KANs)

Kernel Activation Networks (KANs) represent a novel approach to neural network architecture. They challenge the traditional paradigm of fixed activation functions by introducing learnable activation functions on the edges between nodes. This innovation allows for more flexible and potentially more powerful neural networks.

import numpy as np
import matplotlib.pyplot as plt

def traditional_activation(x):
    return np.maximum(0, x)  # ReLU activation

def kan_activation(x, a, b, c):
    return a * np.tanh(b * x) + c  # Learnable activation

x = np.linspace(-5, 5, 100)

plt.figure(figsize=(10, 5))
plt.plot(x, traditional_activation(x), label='Traditional (ReLU)')
plt.plot(x, kan_activation(x, 1.5, 0.5, 0.5), label='KAN (Learnable)')
plt.legend()
plt.title('Traditional vs KAN Activation Functions')
plt.xlabel('Input')
plt.ylabel('Output')
plt.grid(True)
plt.show()

Traditional Neural Networks

Traditional neural networks typically use fixed activation functions applied uniformly across all nodes in a layer. These functions introduce non-linearity, enabling the network to learn complex patterns. Common activation functions include ReLU, sigmoid, and tanh.

import torch
import torch.nn as nn

class TraditionalNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.layers = nn.Sequential(
            nn.Linear(10, 20),
            nn.ReLU(),
            nn.Linear(20, 15),
            nn.ReLU(),
            nn.Linear(15, 1)
        )
    
    def forward(self, x):
        return self.layers(x)

model = TraditionalNN()
print(model)

Kernel Activation Networks (KANs)

KANs redefine the network structure by moving activation functions to the edges and making them learnable parameters. This approach allows each connection to have its own unique, trainable activation function, potentially increasing the network's expressiveness and adaptability.

import torch
import torch.nn as nn

class KernelActivation(nn.Module):
    def __init__(self, in_features, out_features):
        super().__init__()
        self.weight = nn.Parameter(torch.randn(out_features, in_features))
        self.a = nn.Parameter(torch.ones(out_features, in_features))
        self.b = nn.Parameter(torch.ones(out_features, in_features))
        self.c = nn.Parameter(torch.zeros(out_features, in_features))
    
    def forward(self, x):
        linear = torch.matmul(x, self.weight.t())
        return self.a * torch.tanh(self.b * linear) + self.c

class KAN(nn.Module):
    def __init__(self):
        super().__init__()
        self.layers = nn.Sequential(
            KernelActivation(10, 20),
            KernelActivation(20, 15),
            KernelActivation(15, 1)
        )
    
    def forward(self, x):
        return self.layers(x)

model = KAN()
print(model)

Comparison of Traditional NNs and KANs

While traditional neural networks use fixed activation functions, KANs employ learnable activations on edges. This fundamental difference allows KANs to potentially capture more complex relationships in the data, as each connection can adapt its activation function during training.

import torch
import torch.nn as nn
import matplotlib.pyplot as plt

# Traditional NN
traditional_model = nn.Sequential(
    nn.Linear(1, 10),
    nn.ReLU(),
    nn.Linear(10, 1)
)

# KAN (simplified for visualization)
class SimpleKAN(nn.Module):
    def __init__(self):
        super().__init__()
        self.weight = nn.Parameter(torch.randn(1, 10))
        self.a = nn.Parameter(torch.ones(10))
        self.b = nn.Parameter(torch.ones(10))
        self.c = nn.Parameter(torch.zeros(10))
        self.final = nn.Linear(10, 1)
    
    def forward(self, x):
        x = self.a * torch.tanh(self.b * x @ self.weight.t()) + self.c
        return self.final(x)

kan_model = SimpleKAN()

# Visualize
x = torch.linspace(-5, 5, 200).unsqueeze(1)
traditional_out = traditional_model(x).detach()
kan_out = kan_model(x).detach()

plt.figure(figsize=(10, 5))
plt.plot(x, traditional_out, label='Traditional NN')
plt.plot(x, kan_out, label='KAN')
plt.legend()
plt.title('Output Comparison: Traditional NN vs KAN')
plt.xlabel('Input')
plt.ylabel('Output')
plt.grid(True)
plt.show()

Advantages of KANs

KANs offer several potential advantages over traditional neural networks. Their learnable activation functions allow for greater flexibility in modeling complex relationships. This adaptability may lead to improved performance on certain tasks, especially those requiring fine-grained control over the activation landscape.

import torch
import torch.nn as nn
import torch.optim as optim

# Define a complex target function
def target_function(x):
    return torch.sin(x) + 0.5 * torch.cos(2 * x) + 0.2 * torch.sin(5 * x)

# Generate training data
x = torch.linspace(-5, 5, 1000).unsqueeze(1)
y = target_function(x)

# Train both models
traditional_model = nn.Sequential(nn.Linear(1, 20), nn.ReLU(), nn.Linear(20, 1))
kan_model = SimpleKAN()

def train_model(model, x, y, epochs=1000):
    optimizer = optim.Adam(model.parameters())
    criterion = nn.MSELoss()
    for _ in range(epochs):
        optimizer.zero_grad()
        loss = criterion(model(x), y)
        loss.backward()
        optimizer.step()
    return model

traditional_model = train_model(traditional_model, x, y)
kan_model = train_model(kan_model, x, y)

# Visualize results
plt.figure(figsize=(10, 5))
plt.plot(x, y, label='Target Function')
plt.plot(x, traditional_model(x).detach(), label='Traditional NN')
plt.plot(x, kan_model(x).detach(), label='KAN')
plt.legend()
plt.title('Function Approximation: Traditional NN vs KAN')
plt.xlabel('Input')
plt.ylabel('Output')
plt.grid(True)
plt.show()

Challenges and Considerations

While KANs offer potential benefits, they also come with challenges. The increased number of learnable parameters may lead to overfitting on smaller datasets. Additionally, the computational complexity of training KANs can be higher than traditional networks, potentially increasing training time and resource requirements.

import torch
import torch.nn as nn
import torch.optim as optim

def train_and_evaluate(model, x_train, y_train, x_test, y_test, epochs=1000):
    optimizer = optim.Adam(model.parameters())
    criterion = nn.MSELoss()
    train_losses = []
    test_losses = []
    
    for _ in range(epochs):
        # Train
        model.train()
        optimizer.zero_grad()
        train_loss = criterion(model(x_train), y_train)
        train_loss.backward()
        optimizer.step()
        train_losses.append(train_loss.item())
        
        # Evaluate
        model.eval()
        with torch.no_grad():
            test_loss = criterion(model(x_test), y_test)
            test_losses.append(test_loss.item())
    
    return train_losses, test_losses

# Generate data
x = torch.linspace(-5, 5, 1000).unsqueeze(1)
y = torch.sin(x) + 0.1 * torch.randn_like(x)
split = 800
x_train, y_train = x[:split], y[:split]
x_test, y_test = x[split:], y[split:]

# Train both models
traditional_model = nn.Sequential(nn.Linear(1, 20), nn.ReLU(), nn.Linear(20, 1))
kan_model = SimpleKAN()

trad_train, trad_test = train_and_evaluate(traditional_model, x_train, y_train, x_test, y_test)
kan_train, kan_test = train_and_evaluate(kan_model, x_train, y_train, x_test, y_test)

# Visualize learning curves
plt.figure(figsize=(10, 5))
plt.plot(trad_train, label='Traditional Train')
plt.plot(trad_test, label='Traditional Test')
plt.plot(kan_train, label='KAN Train')
plt.plot(kan_test, label='KAN Test')
plt.legend()
plt.title('Learning Curves: Traditional NN vs KAN')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.yscale('log')
plt.grid(True)
plt.show()

Implementation Considerations

Implementing KANs requires careful consideration of initialization strategies, regularization techniques, and optimization algorithms. The choice of the base activation function (e.g., tanh, sigmoid) and the parameterization of learnable components can significantly impact the network's performance and training stability.

import torch
import torch.nn as nn

class AdvancedKernelActivation(nn.Module):
    def __init__(self, in_features, out_features):
        super().__init__()
        self.weight = nn.Parameter(torch.Tensor(out_features, in_features))
        self.a = nn.Parameter(torch.Tensor(out_features, in_features))
        self.b = nn.Parameter(torch.Tensor(out_features, in_features))
        self.c = nn.Parameter(torch.Tensor(out_features, in_features))
        self.reset_parameters()
    
    def reset_parameters(self):
        nn.init.kaiming_uniform_(self.weight, a=math.sqrt(5))
        nn.init.uniform_(self.a, 0.9, 1.1)
        nn.init.uniform_(self.b, 0.9, 1.1)
        nn.init.zeros_(self.c)
    
    def forward(self, x):
        linear = torch.matmul(x, self.weight.t())
        return self.a * torch.tanh(self.b * linear) + self.c

class RegularizedKAN(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super().__init__()
        self.layer1 = AdvancedKernelActivation(input_dim, hidden_dim)
        self.layer2 = AdvancedKernelActivation(hidden_dim, output_dim)
        self.dropout = nn.Dropout(0.5)
    
    def forward(self, x):
        x = self.layer1(x)
        x = self.dropout(x)
        return self.layer2(x)

model = RegularizedKAN(10, 20, 1)
print(model)

# Example of custom loss function with regularization
def custom_loss(output, target, model, l1_lambda=0.01):
    mse_loss = nn.MSELoss()(output, target)
    l1_reg = sum(p.abs().sum() for p in model.parameters())
    return mse_loss + l1_lambda * l1_reg

Real-life Example: Image Classification

KANs can be applied to various machine learning tasks, including image classification. In this example, we'll compare a traditional convolutional neural network (CNN) with a KAN-based CNN for classifying handwritten digits from the MNIST dataset.

import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
from torch.utils.data import DataLoader

# Traditional CNN
class TraditionalCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 32, 3)
        self.conv2 = nn.Conv2d(32, 64, 3)
        self.fc = nn.Linear(1600, 10)
        self.relu = nn.ReLU()
        self.pool = nn.MaxPool2d(2)
    
    def forward(self, x):
        x = self.pool(self.relu(self.conv1(x)))
        x = self.pool(self.relu(self.conv2(x)))
        x = x.view(-1, 1600)
        return self.fc(x)

# KAN-based CNN
class KANCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = KernelActivation(1, 32, kernel_size=3)
        self.conv2 = KernelActivation(32, 64, kernel_size=3)
        self.fc = KernelActivation(1600, 10)
        self.pool = nn.MaxPool2d(2)
    
    def forward(self, x):
        x = self.pool(self.conv1(x))
        x = self.pool(self.conv2(x))
        x = x.view(-1, 1600)
        return self.fc(x)

# Load MNIST dataset
transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))])
train_dataset = datasets.MNIST(root='./data', train=True, download=True, transform=transform)
test_dataset = datasets.MNIST(root='./data', train=False, download=True, transform=transform)
train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=1000, shuffle=False)

# Train and evaluate models
def train_and_evaluate(model, train_loader, test_loader, epochs=5):
    optimizer = optim.Adam(model.parameters())
    criterion = nn.CrossEntropyLoss()
    
    for epoch in range(epochs):
        model.train()
        for batch_idx, (data, target) in enumerate(train_loader):
            optimizer.zero_grad()
            output = model(data)
            loss = criterion(output, target)
            loss.backward()
            optimizer.step()
    
    model.eval()
    correct = 0
    with torch.no_grad():
        for data, target in test_loader:
            output = model(data)
            pred = output.argmax(dim=1, keepdim=True)
            correct += pred.eq(target.view_as(pred)).sum().item()
    
    accuracy = 100. * correct / len(test_loader.dataset)
    return accuracy

traditional_cnn = TraditionalCNN()
kan_cnn = KANCNN()

traditional_accuracy = train_and_evaluate(traditional_cnn, train_loader, test_loader)
kan_accuracy = train_and_evaluate(kan_cnn, train_loader, test_loader)

print(f"Traditional CNN Accuracy: {traditional_accuracy:.2f}%")
print(f"KAN-based CNN Accuracy: {kan_accuracy:.2f}%")

Real-life Example: Natural Language Processing

KANs can also be applied to natural language processing tasks. In this example, we'll compare a traditional recurrent neural network (RNN) with a KAN-based RNN for sentiment analysis on movie reviews.

import torch
import torch.nn as nn
import torch.optim as optim
from torchtext.datasets import IMDB
from torchtext.data.utils import get_tokenizer
from torchtext.vocab import build_vocab_from_iterator

# Simplified model definitions
class TraditionalRNN(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.rnn = nn.RNN(embed_dim, hidden_dim, batch_first=True)
        self.fc = nn.Linear(hidden_dim, 2)
    
    def forward(self, x):
        x = self.embedding(x)
        _, hidden = self.rnn(x)
        return self.fc(hidden.squeeze(0))

class KANRNN(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.kan_layer = KernelActivation(embed_dim, hidden_dim)
        self.fc = nn.Linear(hidden_dim, 2)
    
    def forward(self, x):
        x = self.embedding(x)
        x = self.kan_layer(x.mean(dim=1))  # Simplified KAN layer
        return self.fc(x)

# Data preparation and training (pseudocode)
# 1. Load IMDB dataset
# 2. Tokenize text and build vocabulary
# 3. Create data loaders
# 4. Initialize models, optimizers, and loss function
# 5. Train both models for multiple epochs
# 6. Evaluate models on test set and compare performance

KAN Architecture Variations

KANs can be implemented with various architectural modifications. Some variations include using different base functions for the kernel activations, combining KANs with attention mechanisms, or creating hybrid models that use both traditional and KAN layers.

import torch
import torch.nn as nn

class FlexibleKernelActivation(nn.Module):
    def __init__(self, in_features, out_features, base_function='tanh'):
        super().__init__()
        self.weight = nn.Parameter(torch.randn(out_features, in_features))
        self.a = nn.Parameter(torch.ones(out_features, in_features))
        self.b = nn.Parameter(torch.ones(out_features, in_features))
        self.c = nn.Parameter(torch.zeros(out_features, in_features))
        
        if base_function == 'tanh':
            self.base_func = torch.tanh
        elif base_function == 'sigmoid':
            self.base_func = torch.sigmoid
        elif base_function == 'relu':
            self.base_func = torch.relu
        else:
            raise ValueError("Unsupported base function")
    
    def forward(self, x):
        linear = torch.matmul(x, self.weight.t())
        return self.a * self.base_func(self.b * linear) + self.c

class HybridKANModel(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super().__init__()
        self.traditional_layer = nn.Linear(input_dim, hidden_dim)
        self.kan_layer = FlexibleKernelActivation(hidden_dim, hidden_dim, 'sigmoid')
        self.output_layer = nn.Linear(hidden_dim, output_dim)
    
    def forward(self, x):
        x = torch.relu(self.traditional_layer(x))
        x = self.kan_layer(x)
        return self.output_layer(x)

model = HybridKANModel(10, 20, 2)
print(model)

Training Strategies for KANs

Training KANs effectively may require specialized strategies. These can include careful initialization of the learnable activation parameters, gradual unfreezing of activation functions during training, or using custom learning rate schedules for different parts of the network.

import torch
import torch.nn as nn
import torch.optim as optim

class KANWithGradualUnfreeze(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super().__init__()
        self.layer1 = KernelActivation(input_dim, hidden_dim)
        self.layer2 = KernelActivation(hidden_dim, hidden_dim)
        self.layer3 = KernelActivation(hidden_dim, output_dim)
        
        # Initially freeze activation parameters
        self.freeze_activations()
    
    def freeze_activations(self):
        for param in self.parameters():
            if param.shape != self.layer1.weight.shape:
                param.requires_grad = False
    
    def unfreeze_layer(self, layer_num):
        layer = getattr(self, f'layer{layer_num}')
        for param in layer.parameters():
            param.requires_grad = True
    
    def forward(self, x):
        x = self.layer1(x)
        x = self.layer2(x)
        return self.layer3(x)

# Training loop (pseudocode)
# 1. Initialize model and optimizer
# 2. Train with frozen activations for N epochs
# 3. Unfreeze layer1 activations and train for M epochs
# 4. Unfreeze layer2 activations and train for K epochs
# 5. Unfreeze all activations and fine-tune

Interpreting KAN Behaviors

Understanding the behavior of trained KANs can provide insights into how they process information differently from traditional neural networks. Visualization techniques and analysis of learned activation functions can help interpret KAN decision-making processes.

import torch
import matplotlib.pyplot as plt

def visualize_kan_activations(model, layer_name):
    layer = getattr(model, layer_name)
    a, b, c = layer.a.detach(), layer.b.detach(), layer.c.detach()
    
    x = torch.linspace(-5, 5, 100)
    activations = a * torch.tanh(b * x.unsqueeze(1)) + c
    
    plt.figure(figsize=(10, 5))
    for i in range(min(10, activations.shape[1])):
        plt.plot(x, activations[:, i], label=f'Neuron {i+1}')
    plt.title(f'Learned Activation Functions in {layer_name}')
    plt.xlabel('Input')
    plt.ylabel('Output')
    plt.legend()
    plt.grid(True)
    plt.show()

# Assuming 'model' is a trained KAN
visualize_kan_activations(model, 'layer1')

Future Directions and Research Opportunities

KANs open up new avenues for neural network research. Future work might explore combining KANs with other advanced architectures, developing more efficient training algorithms, or applying KANs to solve complex real-world problems in areas such as computer vision, natural language processing, and reinforcement learning.

# Pseudocode for a potential research direction: KAN-Transformer hybrid

class KANTransformerBlock(nn.Module):
    def __init__(self, d_model, nhead):
        super().__init__()
        self.attention = nn.MultiheadAttention(d_model, nhead)
        self.kan_ffn = KernelActivation(d_model, d_model)
        self.norm1 = nn.LayerNorm(d_model)
        self.norm2 = nn.LayerNorm(d_model)
    
    def forward(self, x):
        attn_output, _ = self.attention(x, x, x)
        x = self.norm1(x + attn_output)
        kan_output = self.kan_ffn(x)
        return self.norm2(x + kan_output)

# Future research tasks:
# 1. Implement and test KAN-Transformer on various NLP tasks
# 2. Compare performance with traditional Transformers
# 3. Analyze the learned activation functions in different layers
# 4. Investigate the impact on model interpretability
# 5. Explore potential benefits in transfer learning scenarios

Additional Resources

For those interested in further exploring Kernel Activation Networks, the following resources provide more in-depth information and research findings:

  1. ArXiv paper: "Kernel Activation Functions in Deep Neural Networks" (arXiv:2009.03663) URL: https://arxiv.org/abs/2009.03663
  2. ArXiv paper: "On the Expressivity of Kernel Activation Networks" (arXiv:2012.14938) URL: https://arxiv.org/abs/2012.14938

These papers offer detailed insights into the theoretical foundations and practical applications of KANs, providing a solid starting point for researchers and practitioners interested in this innovative approach to neural network design.

More in AI applications and systems

  1. Knowledge Graph Enhanced Language Agents for Relevant Recommendations10 min

    Knowledge graphs serve as the backbone of KGLA systems, representing entities and relationships in a structured format. We'll implement a basic knowledge...

  2. Introduction to Activate Functions6 min

    Activation functions are crucial components in neural networks, introducing non-linearity to the model and enabling it to learn complex patterns. They...

  3. Pretrained Hybrids vs Neural Architecture Search for Long Range Tasks10 min

    Pretrained Hybrids with MAD Skills (Mix-And-Distribute) and Neural Architecture Search (NAS) are two approaches for designing efficient neural network...

  4. Introducing Foundation Agents The Next Frontier in AI Decision-Making9 min

    Foundation Agents are an emerging paradigm in AI that combines large language models (LLMs) with planning and decision-making capabilities. They aim to...

  5. Principal Orthogonal Latent Components Analysis Network (POLCA Net)8 min

    Principal Orthogonal Latent Components Analysis Network (POLCA Net) is an innovative approach that aims to extend the capabilities of Principal Component...

  6. Instance-wise LoRA (iLoRA) for Sequential Recommendation9 min

    iLoRA is a novel fine-tuning framework designed to address the challenges of individual variability in user behaviors within sequential recommendation...