PyTorch Debugging & Troubleshooting: AI-Optimized Technical Reference

Critical Error Patterns & Solutions

RuntimeError: mat1 and mat2 shapes cannot be multiplied

Root Cause: Tensor shape mismatch in matrix operations
Failure Impact: Immediate training crash with misleading error location
Common Scenarios:

• Convolutional output fed directly to linear layer without flattening
• Mismatched batch sizes between tensors
  Solution Pattern:

# Essential shape debugging wrapper
def debug_shapes(tensor, name="tensor"):
    print(f"{name}: {tensor.shape}")
    return tensor

# Typical fix for conv-to-linear transition
x = x.view(x.size(0), -1)  # Flatten: [batch, channels, h, w] → [batch, features]

CUDA error: device-side assert triggered

Root Cause: GPU-side operation failure with hidden error details
Failure Impact: Complete training halt with no actionable error message
Debug Strategy: Force CPU execution to reveal actual error

if torch.cuda.is_available():
    try:
        result = model(batch.cuda())
    except RuntimeError as e:
        if "device-side assert" in str(e):
            model_cpu = model.cpu()
            batch_cpu = batch.cpu()
            result = model_cpu(batch_cpu)  # Real error revealed

Common Causes (90% of cases):

• Invalid class indices in loss functions
• NaN values in loss computation
• Negative tensor indices

Expected all tensors to be on the same device

Root Cause: CPU/GPU tensor mixing
Prevention Pattern:

def forward(self, x):
    device = next(self.parameters()).device
    x = x.to(device)
    return self.layers(x)

Memory Management & Leak Detection

Memory Leak Patterns

Critical Failure Mode: Gradual memory growth causing OOM after hours of training
Detection Threshold: >100MB growth over 1000 batches indicates leak
Implementation:

class MemoryTracker:
    def __init__(self):
        self.start_memory = torch.cuda.memory_allocated()
        
    def check_memory_leak(self, tolerance_mb=100):
        gc.collect()
        torch.cuda.empty_cache()
        current_memory = torch.cuda.memory_allocated()
        leak_mb = (current_memory - self.start_memory) / 1024**2
        return leak_mb > tolerance_mb

Common Memory Leak Sources

1. Storing full loss tensors: losses.append(loss) → losses.append(loss.item())
2. Missing gradient clearing: Always call optimizer.zero_grad()
3. Validation mode computation graphs: Use torch.no_grad() context

Memory Profiling Configuration

Working Setup (PyTorch 1.13+):

with torch.profiler.profile(
    activities=[torch.profiler.ProfilerActivity.CUDA],
    profile_memory=True,
    # AVOID: record_shapes=True (adds memory overhead)
    # AVOID: with_stack=True (crashes with custom datasets)
) as prof:
    # Training steps

Gradient Flow Debugging

Gradient Monitoring System

Implementation for vanishing/exploding detection:

def register_gradient_hooks(model):
    def hook_fn(module, grad_input, grad_output):
        if grad_output[0] is not None:
            grad_norm = grad_output[0].norm().item()
            if grad_norm > 10 or grad_norm != grad_norm:  # NaN check
                print(f"Gradient issue in {module.__class__.__name__}: norm={grad_norm}")
    
    for name, module in model.named_modules():
        if len(list(module.children())) == 0:  # Leaf modules only
            module.register_backward_hook(hook_fn)

Gradient Explosion Mitigation

Critical Threshold: Gradient norms >100 indicate explosion
Nuclear Option: torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
Learning Rate Adjustment: Halve learning rate until explosion stops

Deterministic Debugging Configuration

Complete Determinism Setup

Performance Impact: 20-30% training slowdown
Use Case: Bug reproduction and consistent debugging sessions

def set_deterministic_mode(seed=42):
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    torch.use_deterministic_algorithms(True)  # PyTorch 1.12+

Tensor Shape Validation System

Strategic Assertion Placement

Philosophy: Catch shape errors at source, not 20 layers downstream

def assert_shape(tensor, expected_shape, name="tensor"):
    if tensor.shape != torch.Size(expected_shape):
        raise ValueError(f"{name} has shape {tensor.shape}, expected {torch.Size(expected_shape)}")

# Usage pattern
def forward(self, x):
    batch_size = x.size(0)
    assert_shape(x, (batch_size, 3, 224, 224), "input")
    features = self.backbone(x)
    assert_shape(features, (batch_size, 512), "features")

Performance Debugging Specifications

DataLoader Optimization Configuration

High-performance setup:

dataloader = DataLoader(
    dataset,
    batch_size=32,
    num_workers=8,          # More workers = faster (usually)
    pin_memory=True,        # Faster GPU transfer  
    persistent_workers=True # Keeps workers alive between epochs
)

Model Benchmarking Protocol

Performance Threshold: >100ms per forward pass indicates bottleneck

def benchmark_model(model, batch, warmup=10, runs=100):
    # Warmup phase
    for _ in range(warmup):
        _ = model(batch)
    torch.cuda.synchronize()
    
    # Timing measurement
    start = time.time()
    for _ in range(runs):
        output = model(batch)
        torch.cuda.synchronize()
    avg_time = (time.time() - start) / runs
    
    return avg_time * 1000  # Return in milliseconds

Distributed Training Debugging

Communication Verification

Critical Test: All-reduce operation verification across ranks

def check_distributed_setup():
    rank = dist.get_rank()
    world_size = dist.get_world_size()
    
    tensor = torch.ones(1).cuda() * rank
    dist.all_reduce(tensor, op=dist.ReduceOp.SUM)
    expected = sum(range(world_size))
    
    if tensor.item() != expected:
        raise RuntimeError(f"All-reduce broken. Got {tensor.item()}, expected {expected}")

Common Distributed Failures

1. NCCL timeout: Network issues or rank desynchronization
2. Hanging on all_reduce: One rank died without notification
3. Different loss values: Data loading inconsistency across ranks
4. Rank 0 OOM: Uneven batch distribution or extra work on master rank

Advanced Memory Analysis

Memory Snapshot Configuration (PyTorch 1.12+)

Correct API usage:

torch.cuda.memory._record_memory_history()  # No parameters
# Run training
torch.cuda.memory._dump_snapshot("memory_snapshot.pickle")
torch.cuda.memory._record_memory_history(None)  # Disable

Hook-Based Debugging System

Dead neuron detection threshold: >90% zero activations
NaN propagation monitoring:

def register_debug_hooks(model):
    def forward_hook(name):
        def hook(module, input, output):
            if torch.isnan(output).any():
                print(f"NaN detected in {name}")
            
            # Dead ReLU detection
            if 'relu' in name.lower() and isinstance(output, torch.Tensor):
                dead_pct = (output == 0).float().mean().item()
                if dead_pct > 0.9:
                    print(f"WARNING: {name} has {dead_pct*100:.1f}% dead neurons")
        return hook
    
    for name, module in model.named_modules():
        if len(list(module.children())) == 0:
            module.register_forward_hook(forward_hook(name))

Debugging Tool Comparison Matrix

Method	Setup Time	Performance Impact	Info Depth	Best Use Case
Print Statements	Immediate	Minimal	Low	Shape/value debugging
torch.profiler	5-10 minutes	Medium (20-30%)	Very High	Memory/performance analysis
Tensor Hooks	2-5 minutes	Low-Medium	Medium	Gradient flow analysis
Memory Snapshots	10-15 minutes	Low	Very High	Persistent leak investigation
Deterministic Mode	1 minute	High (30-50%)	N/A	Bug reproduction
Custom Assertions	1-2 minutes	Minimal	Low	Runtime validation

Critical Resource Requirements

Hardware Specifications for Debugging

• Memory overhead: Profiling adds 10-20% memory usage
• Compute overhead: Deterministic mode reduces throughput by 30-50%
• Storage requirements: Memory snapshots can be 1-10GB
• Network bandwidth: Distributed debugging requires stable high-bandwidth connections

Version Compatibility Matrix

• PyTorch 1.11.x: Profiler crashes with custom dataloaders
• PyTorch 1.12.0: Memory profiler gives incorrect readings
• PyTorch 1.12+: Memory snapshots available, deterministic algorithms supported
• PyTorch 1.13.1: Improved memory leak detection, stable profiling

Production Deployment Considerations

Debug Mode Performance Impact

• Development: Use full debugging suite
• Staging: Enable memory tracking only
• Production: Disable all debugging except critical error logging

Monitoring Thresholds for Production

• Memory growth: >5% per hour indicates potential leak
• Gradient norms: >50 suggests instability
• Training speed: >2x slowdown indicates bottleneck
• Loss divergence: NaN or >10x increase requires immediate intervention

Emergency Debugging Protocols

When Training Fails Catastrophically

1. Immediate: Switch to CPU mode to get real error messages
2. Within 5 minutes: Enable memory tracking and gradient monitoring
3. Within 15 minutes: Set deterministic mode for reproduction
4. Within 30 minutes: Deploy full profiling suite if issue persists

Data Quality Verification

# Essential dataset testing before training
dataset = MyDataset()
for i in range(min(5, len(dataset))):
    try:
        sample = dataset[i]
        print(f"Sample {i} shapes: {[x.shape for x in sample]}")
    except Exception as e:
        raise RuntimeError(f"Dataset broken at index {i}: {e}")

This technical reference provides actionable debugging protocols with specific thresholds, configurations, and decision trees for systematic PyTorch troubleshooting.