xbe.at

Computer vision

Deep Dive into Image Segmentation with SAM

7 min read, 14 sections

Contents
  1. Understanding SAM Architecture
  2. Vision Transformer Implementation
  3. Prompt Encoder Design
  4. Mask Decoder Architecture
  5. Training Data Generation Pipeline
  6. Mask Generation and IoU Prediction
  7. Zero-Shot Segmentation Pipeline
  8. Attention Mechanism Implementation
  9. Loss Functions for Training
  10. Real-world Example: Medical Image Segmentation
  11. Real-world Example: Autonomous Driving Scene Segmentation
  12. Results Analysis Implementation
  13. Dynamic Prompt Generation
  14. Additional Resources

Understanding SAM Architecture

The Segment Anything Model (SAM) introduces a novel architecture combining three main components: an image encoder based on Vision Transformers (ViT), a flexible prompt encoder supporting various input types, and a mask decoder that generates high-quality segmentation masks.

import torch
import torch.nn as nn

class SAMArchitecture(nn.Module):
    def __init__(self, img_size=1024, patch_size=16, in_channels=3):
        super().__init__()
        self.img_encoder = VisionTransformer(
            img_size=img_size,
            patch_size=patch_size,
            in_channels=in_channels,
            embed_dim=1280
        )
        self.prompt_encoder = PromptEncoder()
        self.mask_decoder = MaskDecoder()
        
    def forward(self, image, prompts):
        # Image encoding
        img_features = self.img_encoder(image)
        # Prompt encoding
        prompt_embeddings = self.prompt_encoder(prompts)
        # Mask generation
        masks, iou_pred = self.mask_decoder(img_features, prompt_embeddings)
        return masks, iou_pred

Vision Transformer Implementation

The image encoder utilizes a Vision Transformer architecture modified for dense prediction tasks. It processes the input image in patches and generates a rich feature representation through self-attention mechanisms.

class VisionTransformer(nn.Module):
    def __init__(self, img_size, patch_size, in_channels, embed_dim):
        super().__init__()
        self.patch_embed = PatchEmbedding(
            img_size=img_size,
            patch_size=patch_size,
            in_channels=in_channels,
            embed_dim=embed_dim
        )
        self.pos_embed = nn.Parameter(torch.zeros(1, 
            (img_size // patch_size) ** 2, embed_dim))
        self.transformer_blocks = nn.ModuleList([
            TransformerBlock(embed_dim=embed_dim)
            for _ in range(32)  # SAM uses 32 transformer blocks
        ])
        
    def forward(self, x):
        # Patch embedding
        x = self.patch_embed(x)
        # Add positional embedding
        x = x + self.pos_embed
        # Pass through transformer blocks
        for block in self.transformer_blocks:
            x = block(x)
        return x

Prompt Encoder Design

The prompt encoder is designed to handle multiple input types including points, boxes, and text descriptions. It converts these diverse prompts into a unified embedding space that can be processed by the mask decoder.

class PromptEncoder(nn.Module):
    def __init__(self, embed_dim=256):
        super().__init__()
        self.point_embedder = PointEmbedder(embed_dim)
        self.box_embedder = BoxEmbedder(embed_dim)
        self.text_embedder = TextEmbedder(embed_dim)
        
    def forward(self, prompts):
        embeddings = []
        for prompt in prompts:
            if prompt['type'] == 'point':
                embed = self.point_embedder(prompt['coords'])
            elif prompt['type'] == 'box':
                embed = self.box_embedder(prompt['coords'])
            elif prompt['type'] == 'text':
                embed = self.text_embedder(prompt['text'])
            embeddings.append(embed)
        return torch.stack(embeddings)

Mask Decoder Architecture

The mask decoder takes the image features and prompt embeddings to generate accurate segmentation masks. It uses a transformer-based architecture with cross-attention mechanisms to combine the multimodal inputs.

class MaskDecoder(nn.Module):
    def __init__(self, transformer_dim=256, num_multimask_outputs=3):
        super().__init__()
        self.transformer = DecoderTransformer(
            depth=2,
            embedding_dim=transformer_dim,
            num_heads=8
        )
        self.iou_prediction_head = nn.Sequential(
            nn.Linear(transformer_dim, transformer_dim),
            nn.ReLU(),
            nn.Linear(transformer_dim, num_multimask_outputs)
        )
        self.mask_prediction_head = MaskMLPDecoder(
            transformer_dim=transformer_dim,
            num_multimask_outputs=num_multimask_outputs
        )
    
    def forward(self, image_embeddings, prompt_embeddings):
        # Process embeddings through transformer
        hidden_states = self.transformer(image_embeddings, prompt_embeddings)
        # Generate masks and IoU predictions
        masks = self.mask_prediction_head(hidden_states)
        iou_pred = self.iou_prediction_head(hidden_states)
        return masks, iou_pred

Training Data Generation Pipeline

This implementation shows how SAM generates its massive training dataset using a combination of automated and human-in-the-loop processes to create diverse and high-quality segmentation masks.

import numpy as np
from PIL import Image

class DatasetGenerator:
    def __init__(self, sam_model):
        self.sam_model = sam_model
        self.annotators = []
        
    def generate_masks(self, image_path):
        image = Image.open(image_path)
        # Generate automatic proposals
        proposals = self.generate_proposals(image)
        # Human verification and refinement
        verified_masks = self.human_verification(proposals)
        # Additional prompt-based masks
        prompt_masks = self.generate_prompt_masks(image)
        return self.combine_masks(verified_masks, prompt_masks)
    
    def generate_proposals(self, image):
        # Convert image to tensor
        img_tensor = self.preprocess_image(image)
        # Generate automatic mask proposals
        with torch.no_grad():
            masks = self.sam_model.generate_proposals(img_tensor)
        return masks

Mask Generation and IoU Prediction

The mask generation process combines positional encodings with learned embeddings to predict multiple possible segmentation masks. The IoU prediction head estimates the quality of each generated mask to enable automatic mask selection.

class MaskMLPDecoder(nn.Module):
    def __init__(self, transformer_dim, hidden_dim=256):
        super().__init__()
        self.mlp = nn.Sequential(
            nn.Linear(transformer_dim, hidden_dim),
            nn.GELU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.GELU(),
            nn.Linear(hidden_dim, 1)  # Single channel mask output
        )
        
    def forward(self, x):
        # Reshape features
        b, n, c = x.shape
        x = x.reshape(b * n, c)
        # Generate mask logits
        masks = self.mlp(x)
        # Reshape back to batch format
        masks = masks.reshape(b, n, 1)
        return masks

Zero-Shot Segmentation Pipeline

The implementation demonstrates how SAM handles zero-shot segmentation tasks by processing arbitrary prompts and generating corresponding masks without task-specific training.

class ZeroShotSegmentation:
    def __init__(self, sam_model):
        self.model = sam_model
        
    def segment_image(self, image, prompt):
        # Preprocess image and prompt
        processed_image = self.preprocess_image(image)
        encoded_prompt = self.encode_prompt(prompt)
        
        with torch.no_grad():
            # Generate mask predictions
            masks, scores = self.model(processed_image, encoded_prompt)
            # Select best mask based on IoU prediction
            best_mask_idx = torch.argmax(scores)
            best_mask = masks[best_mask_idx]
            
        return self.postprocess_mask(best_mask)
    
    def preprocess_image(self, image):
        # Convert to tensor and normalize
        transform = transforms.Compose([
            transforms.Resize((1024, 1024)),
            transforms.ToTensor(),
            transforms.Normalize(mean=[0.485, 0.456, 0.406],
                               std=[0.229, 0.224, 0.225])
        ])
        return transform(image).unsqueeze(0)

Attention Mechanism Implementation

The attention mechanism is crucial for combining image and prompt features effectively. This implementation shows the multi-head attention used in both the image encoder and mask decoder.

class MultiheadAttention(nn.Module):
    def __init__(self, embed_dim, num_heads):
        super().__init__()
        self.embed_dim = embed_dim
        self.num_heads = num_heads
        self.head_dim = embed_dim // num_heads
        
        self.qkv = nn.Linear(embed_dim, 3 * embed_dim)
        self.proj = nn.Linear(embed_dim, embed_dim)
        
    def forward(self, x):
        batch_size, seq_len, _ = x.shape
        
        # Generate Q, K, V
        qkv = self.qkv(x)
        qkv = qkv.reshape(batch_size, seq_len, 3, self.num_heads, self.head_dim)
        q, k, v = qkv.unbind(2)
        
        # Compute attention
        attn = (q @ k.transpose(-2, -1)) / math.sqrt(self.head_dim)
        attn = F.softmax(attn, dim=-1)
        
        # Apply attention to values
        x = (attn @ v).transpose(1, 2).reshape(batch_size, seq_len, self.embed_dim)
        x = self.proj(x)
        return x

Loss Functions for Training

The training process involves multiple loss components including mask IoU loss, binary cross-entropy for mask prediction, and auxiliary losses for prompt learning.

class SAMLoss(nn.Module):
    def __init__(self, iou_weight=1.0, mask_weight=1.0):
        super().__init__()
        self.iou_weight = iou_weight
        self.mask_weight = mask_weight
        self.bce_loss = nn.BCEWithLogitsLoss()
        
    def forward(self, pred_masks, pred_iou, target_masks, target_iou):
        # Mask loss using BCE
        mask_loss = self.bce_loss(pred_masks, target_masks)
        
        # IoU prediction loss using MSE
        iou_loss = F.mse_loss(pred_iou, target_iou)
        
        # Combine losses
        total_loss = (self.mask_weight * mask_loss + 
                     self.iou_weight * iou_loss)
        
        return {
            'total_loss': total_loss,
            'mask_loss': mask_loss,
            'iou_loss': iou_loss
        }

Real-world Example: Medical Image Segmentation

Implementation of SAM for medical image segmentation, showing how the model can be adapted for specific domains while maintaining zero-shot capabilities.

class MedicalImageSegmentation:
    def __init__(self, sam_model):
        self.model = sam_model
        self.preprocessing = MedicalImagePreprocessing()
        
    def segment_medical_image(self, dicom_image, roi_prompt):
        # Preprocess DICOM image
        processed_image = self.preprocessing.process_dicom(dicom_image)
        
        # Generate anatomical prompts
        anatomical_prompts = self.generate_anatomical_prompts(roi_prompt)
        
        # Perform segmentation
        masks = []
        for prompt in anatomical_prompts:
            mask, iou = self.model(processed_image, prompt)
            masks.append((mask, iou))
            
        # Post-process results
        final_mask = self.post_process_medical_masks(masks)
        return final_mask
    
    def generate_anatomical_prompts(self, roi):
        # Convert ROI description to model-compatible prompts
        anatomical_points = self.get_anatomical_landmarks(roi)
        return [{'type': 'point', 'coords': point} 
                for point in anatomical_points]

Real-world Example: Autonomous Driving Scene Segmentation

The implementation demonstrates SAM's application in autonomous driving scenarios, handling multiple object classes and real-time segmentation requirements with dynamic prompting.

class AutonomousDrivingSegmentation:
    def __init__(self, sam_model, fps_target=30):
        self.model = sam_model
        self.fps_target = fps_target
        self.object_classes = ['vehicle', 'pedestrian', 'road', 'sign']
        
    def process_video_frame(self, frame):
        # Convert frame to tensor
        frame_tensor = self.preprocess_frame(frame)
        
        # Generate dynamic prompts for each object class
        prompts = self.generate_scene_prompts(frame_tensor)
        
        # Parallel segmentation for all classes
        results = {}
        with torch.no_grad():
            for obj_class, prompt in prompts.items():
                mask, confidence = self.model(frame_tensor, prompt)
                results[obj_class] = {
                    'mask': mask,
                    'confidence': confidence
                }
        
        return self.combine_class_masks(results)
    
    def generate_scene_prompts(self, frame):
        prompts = {}
        for obj_class in self.object_classes:
            prompts[obj_class] = {
                'type': 'text',
                'text': f'Segment all {obj_class}s in the scene'
            }
        return prompts

Results Analysis Implementation

This implementation provides comprehensive evaluation metrics for segmentation quality, including IoU scores, boundary precision, and real-time performance metrics.

class SegmentationEvaluator:
    def __init__(self):
        self.metrics = {
            'iou': [],
            'boundary_f1': [],
            'inference_time': [],
            'memory_usage': []
        }
    
    def evaluate_prediction(self, pred_mask, gt_mask, timing_ms):
        # Calculate IoU
        intersection = np.logical_and(pred_mask, gt_mask)
        union = np.logical_or(pred_mask, gt_mask)
        iou = np.sum(intersection) / np.sum(union)
        
        # Calculate boundary F1 score
        boundary_f1 = self.calculate_boundary_f1(pred_mask, gt_mask)
        
        # Store metrics
        self.metrics['iou'].append(iou)
        self.metrics['boundary_f1'].append(boundary_f1)
        self.metrics['inference_time'].append(timing_ms)
        
        return {
            'iou': iou,
            'boundary_f1': boundary_f1,
            'timing_ms': timing_ms
        }
    
    def calculate_boundary_f1(self, pred, gt, tolerance=2):
        pred_boundary = self.get_boundary(pred)
        gt_boundary = self.get_boundary(gt)
        return self._f1_score(pred_boundary, gt_boundary, tolerance)

Dynamic Prompt Generation

Implementation of an adaptive prompt generation system that optimizes prompting strategy based on previous segmentation results and scene complexity.

class DynamicPromptGenerator:
    def __init__(self, history_size=10):
        self.history = deque(maxlen=history_size)
        self.prompt_strategies = {
            'point': self.generate_point_prompt,
            'box': self.generate_box_prompt,
            'text': self.generate_text_prompt
        }
        
    def generate_optimal_prompt(self, image, scene_context):
        # Analyze scene complexity
        complexity_score = self.analyze_scene_complexity(image)
        
        # Select best strategy based on history
        best_strategy = self.select_strategy(complexity_score)
        
        # Generate prompt using selected strategy
        prompt = self.prompt_strategies[best_strategy](image, scene_context)
        
        # Update history
        self.history.append({
            'strategy': best_strategy,
            'complexity': complexity_score,
            'success': None  # To be updated after segmentation
        })
        
        return prompt
    
    def analyze_scene_complexity(self, image):
        # Calculate image features
        edges = cv2.Canny(image, 100, 200)
        complexity = np.mean(edges) / 255.0
        return complexity

Additional Resources

More in Computer vision

  1. Deep Learning's Impact on Computer Vision10 min

    Computer vision has indeed been greatly impacted by deep learning, but it's important to note that it's not the only success story in the field. While deep...

  2. Convolutional Neural Networks on Manifolds with Python10 min

    Convolutional Neural Networks (CNNs) have revolutionized image processing tasks, but their application to non-Euclidean domains like manifolds presents...

  3. Deep Residual Learning for Image Recognition in Python8 min

    Deep Residual Learning is a revolutionary approach in image recognition that addresses the problem of vanishing gradients in deep neural networks. It...

  4. Convolutional Neural Networks and Kernels in Python6 min

    Convolutional Neural Networks (CNNs) have revolutionized the field of computer vision and image processing. They are designed to automatically and adaptively...

  5. Evaluating GANs with Inception Score and FID7 min

    The evaluation of Generative Adversarial Networks requires quantitative metrics to assess the quality and diversity of generated samples. Two fundamental...

  6. Convolutional Neural Network with Python10 min

    Convolutional operations are fundamental building blocks in neural networks, particularly in computer vision tasks. They enable the network to learn spatial...