xbe.at

Data visualisation

Visualizing Dense Data with Hexbin Plots

9 min read, 14 sections

Contents
  1. Introduction to Hexbin Plots
  2. Mathematical Foundation of Hexbin Plots
  3. Custom Color Mapping
  4. Comparing Scatter vs Hexbin
  5. Statistical Analysis with Hexbin Plots
  6. Real-world Example: Climate Data Analysis
  7. Optimizing Hexbin Parameters
  8. Advanced Hexbin Customization
  9. Time Series Analysis with Hexbin
  10. Multivariate Analysis with Hexbin Matrices
  11. Performance Optimization for Large Datasets
  12. Real-world Example: Geographic Data Analysis
  13. Interactive Hexbin Visualization
  14. Additional Resources

Introduction to Hexbin Plots

Hexbin plots offer superior visualization for dense scatter plots by aggregating data points into hexagonal bins. The color intensity of each hexagon represents the density of points within that region, providing clearer insights into data distribution patterns compared to traditional scatter plots.

import numpy as np
import matplotlib.pyplot as plt

# Generate sample data
np.random.seed(42)
x = np.random.normal(0, 1, 100000)
y = np.random.normal(0, 1, 100000)

# Create hexbin plot
plt.figure(figsize=(10, 8))
plt.hexbin(x, y, gridsize=30, cmap='YlOrRd')
plt.colorbar(label='Count')
plt.xlabel('X-axis')
plt.ylabel('Y-axis')
plt.title('Basic Hexbin Plot')
plt.show()

Mathematical Foundation of Hexbin Plots

The hexagonal binning process involves partitioning the plane into regular hexagons and counting points within each hexagon. The count determines the color intensity using a mapping function f(n), where n is the number of points in each hexagon.

# Mathematical representation of hexbin calculation
'''
Hexagon center coordinates (x,y) calculation:
$$x = (i + 0.5 * (j\%2)) * w$$
$$y = j * h * \sqrt{3}/2$$

where:
- i,j are grid indices
- w is hexagon width
- h is hexagon height
'''

# Implementation of hexagonal grid
def create_hex_grid(xmin, xmax, ymin, ymax, gridsize):
    w = (xmax - xmin) / gridsize
    h = (ymax - ymin) / gridsize
    
    x_centers = []
    y_centers = []
    
    for i in range(gridsize):
        for j in range(gridsize):
            x = (i + 0.5 * (j % 2)) * w + xmin
            y = j * h * np.sqrt(3)/2 + ymin
            x_centers.append(x)
            y_centers.append(y)
    
    return np.array(x_centers), np.array(y_centers)

Custom Color Mapping

Advanced hexbin visualization requires careful consideration of color mapping to effectively represent data density. This implementation demonstrates how to create custom colormaps and normalize count values for better visualization.

import matplotlib.colors as colors

# Generate sample data
np.random.seed(42)
x = np.random.exponential(1, 50000)
y = np.random.exponential(1, 50000)

# Create custom colormap
colors_list = ['#f7fbff', '#deebf7', '#c6dbef', '#9ecae1', '#6baed6', '#4292c6', '#2171b5', '#084594']
custom_cmap = colors.LinearSegmentedColormap.from_list('custom', colors_list)

# Create hexbin with custom colors
plt.figure(figsize=(10, 8))
plt.hexbin(x, y, gridsize=30, cmap=custom_cmap, 
           norm=colors.LogNorm(vmin=1, vmax=1000))
plt.colorbar(label='Count (log scale)')
plt.xlabel('X-axis')
plt.ylabel('Y-axis')
plt.title('Hexbin Plot with Custom Colormap')
plt.show()

Comparing Scatter vs Hexbin

Understanding the transformation from scatter to hexbin plots requires direct comparison. This implementation shows both visualizations side by side using the same dataset to highlight the advantages of hexagonal binning.

# Generate dense clustered data
np.random.seed(42)
n_points = 100000
centers = [(0, 0), (2, 2), (-2, -2)]
x = np.concatenate([np.random.normal(cx, 0.5, n_points//3) for cx, _ in centers])
y = np.concatenate([np.random.normal(cy, 0.5, n_points//3) for _, cy in centers])

# Create comparison plot
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 6))

# Scatter plot
ax1.scatter(x, y, alpha=0.1, s=1)
ax1.set_title('Scatter Plot')

# Hexbin plot
hb = ax2.hexbin(x, y, gridsize=30, cmap='YlOrRd')
ax2.set_title('Hexbin Plot')
plt.colorbar(hb, ax=ax2, label='Count')

plt.tight_layout()
plt.show()

Statistical Analysis with Hexbin Plots

Hexbin plots can be enhanced with statistical overlays to provide deeper insights. This implementation adds contour lines representing probability density estimates and statistical markers for cluster centers, combining density visualization with quantitative analysis.

import scipy.stats as stats

# Generate multivariate normal data
np.random.seed(42)
n_samples = 50000
mean = [0, 0]
cov = [[1, 0.5], [0.5, 1]]
data = np.random.multivariate_normal(mean, cov, n_samples)

# Create enhanced hexbin plot
fig, ax = plt.subplots(figsize=(12, 8))

# Plot hexbin
hb = ax.hexbin(data[:, 0], data[:, 1], gridsize=30, cmap='YlOrRd')

# Add kernel density estimation contours
xmin, xmax = ax.get_xlim()
ymin, ymax = ax.get_ylim()
xx, yy = np.mgrid[xmin:xmax:100j, ymin:ymax:100j]
positions = np.vstack([xx.ravel(), yy.ravel()])
kernel = stats.gaussian_kde(data.T)
z = np.reshape(kernel(positions).T, xx.shape)

# Plot contours
ax.contour(xx, yy, z, colors='black', alpha=0.5)

plt.colorbar(hb, label='Count')
plt.title('Hexbin Plot with Statistical Overlay')
plt.xlabel('X-axis')
plt.ylabel('Y-axis')
plt.show()

Real-world Example: Climate Data Analysis

Analyzing climate data patterns using hexbin plots demonstrates their practical application. This example processes temperature and humidity measurements from multiple weather stations, revealing regional climate patterns.

# Simulating weather station data
np.random.seed(42)
n_stations = 100000

# Generate synthetic climate data
temperature = np.random.normal(25, 5, n_stations) + \
             np.sin(np.linspace(0, 2*np.pi, n_stations)) * 2
humidity = temperature * (-0.5) + np.random.normal(60, 10, n_stations)

# Create climate analysis plot
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 6))

# Hexbin plot
hb1 = ax1.hexbin(temperature, humidity, gridsize=30, 
                 cmap='YlOrRd', mincnt=1)
ax1.set_xlabel('Temperature (°C)')
ax1.set_ylabel('Humidity (%)')
ax1.set_title('Temperature vs Humidity Distribution')
plt.colorbar(hb1, ax=ax1, label='Frequency')

# Add marginal distributions
temp_hist = ax2.hist(temperature, bins=50, density=True)
ax2.set_xlabel('Temperature (°C)')
ax2.set_ylabel('Density')
ax2.set_title('Temperature Distribution')

plt.tight_layout()
plt.show()

Optimizing Hexbin Parameters

Fine-tuning hexbin plot parameters is crucial for effective visualization. This implementation demonstrates the impact of different gridsize values and explores various methods for determining optimal bin sizes based on data characteristics.

def optimize_hexbin(x, y, min_grid=20, max_grid=50):
    # Calculate optimal gridsize based on data spread
    x_range = np.ptp(x)
    y_range = np.ptp(y)
    
    # Create comparison plot with different gridsizes
    fig, axs = plt.subplots(2, 2, figsize=(15, 15))
    gridsizes = [20, 30, 40, 50]
    
    for ax, gridsize in zip(axs.flat, gridsizes):
        hb = ax.hexbin(x, y, gridsize=gridsize, cmap='YlOrRd')
        ax.set_title(f'Gridsize: {gridsize}')
        plt.colorbar(hb, ax=ax)
    
    plt.tight_layout()
    plt.show()

# Generate sample data
np.random.seed(42)
x = np.concatenate([np.random.normal(loc, 1, 10000) for loc in [-2, 2]])
y = np.concatenate([np.random.normal(loc, 1, 10000) for loc in [-2, 2]])

# Optimize visualization
optimize_hexbin(x, y)

Advanced Hexbin Customization

Advanced customization techniques enable the creation of sophisticated visualizations that combine hexbin plots with additional statistical information and custom styling options for enhanced data interpretation.

import seaborn as sns
from matplotlib.patches import RegularPolygon

# Generate correlated data
np.random.seed(42)
n_points = 50000
x = np.random.normal(0, 1, n_points)
y = x * 0.5 + np.random.normal(0, 0.5, n_points)

# Create enhanced hexbin plot
fig, ax = plt.subplots(figsize=(12, 8))

# Custom colormap
custom_cmap = sns.cubehelix_palette(as_cmap=True, reverse=True)

# Create hexbin with customization
hb = ax.hexbin(x, y, gridsize=30, cmap=custom_cmap,
               linewidths=0.2, edgecolors='white',
               norm=colors.LogNorm(vmin=1, vmax=1000))

# Add statistical annotations
pearson_corr = np.corrcoef(x, y)[0, 1]
ax.text(0.05, 0.95, f'Correlation: {pearson_corr:.2f}',
        transform=ax.transAxes, bbox=dict(facecolor='white', alpha=0.8))

plt.colorbar(hb, label='Count (log scale)')
plt.title('Enhanced Hexbin Visualization')
plt.xlabel('X-axis')
plt.ylabel('Y-axis')
plt.show()

Time Series Analysis with Hexbin

Hexbin plots can effectively visualize temporal patterns in large datasets. This implementation shows how to analyze time series data by combining hexbin visualization with temporal decomposition to reveal underlying patterns and anomalies.

import pandas as pd
from datetime import datetime, timedelta

# Generate time series data
np.random.seed(42)
dates = pd.date_range(start='2023-01-01', end='2023-12-31', freq='H')
n_points = len(dates)

# Create synthetic temporal patterns
base_signal = np.sin(np.linspace(0, 4*np.pi, n_points)) * 10
daily_pattern = np.sin(np.linspace(0, 2*np.pi*n_points/24, n_points)) * 5
noise = np.random.normal(0, 2, n_points)
values = base_signal + daily_pattern + noise

# Convert to hours of day vs. values
hours = dates.hour.values
days = dates.dayofyear.values

plt.figure(figsize=(12, 8))
plt.hexbin(hours, values, gridsize=30, cmap='viridis')
plt.colorbar(label='Frequency')
plt.xlabel('Hour of Day')
plt.ylabel('Value')
plt.title('Time Series Patterns Using Hexbin')

# Add rolling average overlay
hour_avg = pd.Series(values).groupby(hours).mean()
plt.plot(range(24), hour_avg, 'r-', linewidth=2, label='Hourly Average')
plt.legend()
plt.show()

Multivariate Analysis with Hexbin Matrices

When dealing with multiple variables, creating a matrix of hexbin plots can reveal complex relationships. This implementation demonstrates how to create a hexbin correlation matrix for multivariate analysis.

def hexbin_matrix(data, var_names, gridsize=30):
    n_vars = len(var_names)
    fig, axes = plt.subplots(n_vars, n_vars, figsize=(15, 15))
    
    for i in range(n_vars):
        for j in range(n_vars):
            if i != j:
                hb = axes[i,j].hexbin(data[:,j], data[:,i], 
                                    gridsize=gridsize,
                                    cmap='YlOrRd')
                if i == n_vars-1:
                    axes[i,j].set_xlabel(var_names[j])
                if j == 0:
                    axes[i,j].set_ylabel(var_names[i])
            else:
                axes[i,j].hist(data[:,i], bins=30)
                axes[i,j].set_xlabel(var_names[i])
    
    plt.tight_layout()
    return fig, axes

# Generate multivariate data
n_samples = 10000
var_names = ['X1', 'X2', 'X3', 'X4']
data = np.random.multivariate_normal(
    mean=[0, 0, 0, 0],
    cov=[[1, .5, .3, .2],
         [.5, 1, .4, .1],
         [.3, .4, 1, .6],
         [.2, .1, .6, 1]],
    size=n_samples
)

# Create hexbin matrix
hexbin_matrix(data, var_names)
plt.show()

Performance Optimization for Large Datasets

When working with massive datasets, optimizing hexbin plot performance becomes crucial. This implementation shows techniques for efficient data handling and visualization of millions of points.

import time
import datashader as ds
import colorcet as cc

def optimized_hexbin(x, y, n_points, method='standard'):
    data = pd.DataFrame({
        'x': x[:n_points],
        'y': y[:n_points]
    })
    
    start_time = time.time()
    
    if method == 'standard':
        plt.figure(figsize=(10, 8))
        plt.hexbin(data['x'], data['y'], gridsize=50, cmap='YlOrRd')
    
    elif method == 'datashader':
        cvs = ds.Canvas(plot_width=400, plot_height=400)
        agg = cvs.hexbin(data, 'x', 'y')
        ds.tf.shade(agg, cmap=cc.fire)
    
    end_time = time.time()
    print(f"Method: {method}, Time: {end_time - start_time:.2f} seconds")

# Generate large dataset
np.random.seed(42)
n_points = 1000000
x = np.random.normal(0, 1, n_points)
y = np.random.normal(0, 1, n_points)

# Compare methods
optimized_hexbin(x, y, n_points, 'standard')
optimized_hexbin(x, y, n_points, 'datashader')
plt.show()

Real-world Example: Geographic Data Analysis

Hexbin plots excel at visualizing geographic point data, especially for analyzing population density or event distributions across regions. This implementation shows how to create location-based hexbin visualizations with proper coordinate handling.

import geopandas as gpd
from shapely.geometry import Point

# Generate synthetic geographic data (lat/long points)
np.random.seed(42)
n_points = 50000

# Simulate points in a geographic region (e.g., city boundaries)
lats = np.random.normal(40.7128, 0.1, n_points)  # New York City approx
lons = np.random.normal(-74.0060, 0.1, n_points)

# Create the plot
fig, ax = plt.subplots(figsize=(12, 8))

# Create hexbin with geographic coordinates
hb = ax.hexbin(lons, lats, 
               gridsize=30,
               cmap='YlOrRd',
               mincnt=1,
               bins='log')

# Add proper geographic formatting
ax.set_xlabel('Longitude')
ax.set_ylabel('Latitude')
plt.colorbar(hb, label='Log10(Count)')

# Add title and gridlines
ax.set_title('Geographic Point Density Analysis')
ax.grid(True, alpha=0.3)

# Optional: Add background map context
try:
    world = gpd.read_file(gpd.datasets.get_path('naturalearth_lowres'))
    world.plot(ax=ax, alpha=0.1, color='grey')
except:
    pass

plt.show()

Interactive Hexbin Visualization

Modern data analysis requires interactive visualizations. This implementation creates an interactive hexbin plot using Plotly, enabling zoom, pan, and hover functionality for detailed exploration.

import plotly.graph_objects as go
import plotly.express as px

# Generate sample data
np.random.seed(42)
n_points = 50000
x = np.random.normal(0, 1, n_points)
y = x * 0.5 + np.random.normal(0, 0.5, n_points)

# Create interactive hexbin
fig = go.Figure()

# Add hexbin layer
fig.add_trace(go.Histogram2d(
    x=x,
    y=y,
    nbinsx=30,
    nbinsy=30,
    colorscale='Viridis',
    histnorm='probability',
    showscale=True
))

# Update layout
fig.update_layout(
    title='Interactive Hexbin Analysis',
    xaxis_title='X Axis',
    yaxis_title='Y Axis',
    width=800,
    height=600,
    showlegend=False
)

# Add hover template
fig.update_traces(
    hoverongaps=False,
    hovertemplate='x: %{x}<br>y: %{y}<br>density: %{z}<extra></extra>'
)

# Show the interactive plot
fig.show()

Additional Resources

More in Data visualisation

  1. Visualizing Galois Theory with Python8 min

    Galois Theory is a branch of abstract algebra that provides a connection between field theory and group theory. It was developed by Évariste Galois in the...

  2. Visualizing Data Beyond Bar and Line Charts11 min

    Size-encoded heatmaps improve upon traditional heatmaps by adding a size component to each cell. This enhancement makes it easier to interpret exact values...

  3. Visualizing High-Dimensional Data with Stochastic Neighbor Embedding9 min

    Stochastic Neighbor Embedding (SNE) is a machine learning algorithm for dimensionality reduction, particularly useful for visualizing high-dimensional data...

  4. Visualizing Complex Analysis with Python6 min

    Complex analysis is a branch of mathematics that studies functions of complex numbers. It has wide-ranging applications in physics, engineering, and other...

  5. Visualizing High-Dimensional Data with t-SNE10 min

    t-Distributed Stochastic Neighbor Embedding (t-SNE) is a machine learning algorithm for visualization of high-dimensional data. It reduces dimensionality...

  6. Visualizing Classifier Performance with ROC Plots11 min

    ROC (Receiver Operating Characteristic) plots are powerful tools for evaluating binary classification models. They visualize the trade-off between true...