CUDA Production Debugging: AI-Optimized Knowledge Base

Critical Configuration Settings

Essential Environment Variables for Production Debugging

# Mandatory debugging setup
export CUDA_LAUNCH_BLOCKING=1          # Forces synchronous execution
export CUDA_ENABLE_COREDUMP_ON_EXCEPTION=1  # Captures GPU state on crashes
export CUDA_COREDUMP_FILE=/tmp/cuda_crash_%p_%t

# Memory debugging
export CUDA_MEMCHECK=1                 # Enables memory checking
export CUDA_DEVICE_MAX_CONNECTIONS=1   # Serializes kernel launches

# Extreme debugging (performance impact: 50x slower)
export CUDA_DEVICE_WAITS_ON_EXCEPTION=1

Hardware Compatibility Requirements

• Core dumps: Only work on Linux with Tesla, Quadro, and RTX cards
• GeForce cards: Require special driver flags that may void warranty
• compute-sanitizer: Causes 10-50x performance degradation

Critical Failure Patterns

Asynchronous Error Reporting (Severity: Critical)

Problem: CUDA reports errors 3+ kernel launches after actual failure
Impact: Python stacktraces point to completely unrelated code
Detection: Error messages suggest CUDA_LAUNCH_BLOCKING=1 but this fails with CUDA graphs
Solution: Use core dumps - they capture exact failure point at hardware level

Memory Access Violations (Frequency: Most Common)

Buffer Overflows

// Crash pattern
__global__ void broken_kernel(float* data, int size) {
    int idx = blockIdx.x * blockDim.x + threadIdx.x;
    data[idx] = 42.0f; // Crashes when idx >= size
}

Critical Fix: Always check bounds before memory access

Double-Free Detection

cudaFree(ptr);
// 500 lines later...
cudaFree(ptr); // Illegal memory access

Behavior: Sometimes works, sometimes crashes depending on GPU memory allocator state
Detection Method: Use cudaDeviceSynchronize() after frees during debugging

Use-After-Free

cudaFree(device_buffer);
kernel<<<blocks, threads>>>(device_buffer); // Accessing freed memory

Debugging Tools Performance Characteristics

compute-sanitizer (Primary Tool)

compute-sanitizer --tool=memcheck ./app      # Buffer overflows, use-after-free
compute-sanitizer --tool=racecheck ./app     # Race conditions between threads
compute-sanitizer --tool=initcheck ./app     # Uninitialized memory reads

Performance Impact: 10-50x slower execution
Recommendation: Use only on small test cases, not full datasets

cuda-gdb (Secondary Tool)

cuda-gdb ./app
(cuda-gdb) set cuda memcheck on
(cuda-gdb) info cuda kernels    # Show running kernels
(cuda-gdb) cuda thread          # Switch between GPU threads
(cuda-gdb) cuda block           # Examine specific thread blocks

Reliability Warning: cuda-gdb crashes more frequently than target applications
Use Case: Only tool providing actual GPU stack traces when functional

Production Failure Scenarios

Memory Leak Disguised as Framework Issue

Symptom: PyTorch consuming 32GB GPU memory/hour until OOM
Root Cause: PyTorch caching allocator holding freed memory
Detection: torch.cuda.memory_allocated() constant, torch.cuda.memory_reserved() growing
Solution: Periodic cleanup every 100 iterations, not every iteration

Race Condition Heisenbug (0.1% failure rate)

Symptom: Matrix multiplication correct 99.9% of time, wrong only in production
Root Cause: Bank conflicts in shared memory under concurrent kernel load
Detection: Nsight Compute "Bank Conflict Rate" >12%
Fix: Change shared memory access from row-major to column-major pattern

Driver Version Conflicts

Symptom: CUDA_ERROR_SYSTEM_DRIVER_MISMATCH after routine updates
Root Cause: Ubuntu unattended upgrades install nouveau drivers conflicting with NVIDIA
Critical Fix: Always pin kernel and driver versions in production

# Emergency recovery
lsmod | grep nouveau
echo 'blacklist nouveau' > /etc/modprobe.d/blacklist-nouveau.conf
update-initramfs -u

Thermal Throttling (Performance Degradation)

Symptom: CUDA kernels progressively slower, no crashes
Detection: GPU temperature >87°C causes throttling from 1.7GHz to 800MHz
Impact: 50% performance loss without application crashes
Monitoring: nvidia-smi --query-gpu=temperature.gpu,clocks.current.graphics

ECC Memory Errors (Hardware Failure)

Pattern: Random CUDA_ERROR_ECC_UNCORRECTABLE after 6-8 hours
Detection: nvidia-smi -q -d ECC | grep "Double Bit ECC Errors"
Critical: Double-bit errors indicate dying GPU memory - RMA required
Warning: Single-bit errors corrected automatically but indicate impending failure

Multi-GPU Deadlocks

Symptom: Training hangs indefinitely during collective operations
Root Cause: NCCL communication deadlock from unsynchronized GPU streams
Fix Pattern: Separate kernel launches from synchronization

// Wrong: Can cause deadlocks
for (int gpu = 0; gpu < num_gpus; gpu++) {
    cudaSetDevice(gpu);
    kernel<<<...>>>();
    cudaStreamSynchronize(stream);
}

// Correct: Synchronize all streams together
for (int gpu = 0; gpu < num_gpus; gpu++) {
    cudaSetDevice(gpu);
    kernel<<<...>>>();
}
for (int gpu = 0; gpu < num_gpus; gpu++) {
    cudaSetDevice(gpu);
    cudaStreamSynchronize(stream);
}

Resource Requirements and Time Costs

Debugging Tool Time Investment

• compute-sanitizer: 50x execution time increase, requires separate test datasets
• cuda-gdb: 2-4 hours learning curve, frequent crashes extend debugging sessions
• Core dump analysis: 30 minutes setup, saves hours of async error hunting
• Driver reinstallation: 1-2 hours downtime, requires system reboot

Production Debugging Prerequisites

• Identical hardware: Essential for driver version matching
• Temperature monitoring: Continuous GPU temperature tracking required
• ECC error tracking: Regular memory health checks prevent cascading failures
• Framework-specific tools: PyTorch memory profiling separate from nvidia-smi

Hardware-Level Debugging Approach

GPU Memory Fragmentation

Detection: cudaMalloc() fails despite sufficient total memory
Impact: Large allocation failures in long-running applications
Solutions by effectiveness:

1. Memory pools with cudaMemPool* APIs (CUDA 11.2+)
2. Early large block allocation with manual sub-allocation
3. cudaDeviceReset() (destroys all allocations - nuclear option)

Bank Conflicts Performance Impact

Detection: Nsight Compute "Shared Memory Bank Conflicts per Request" >1.0
Performance Impact: Can reduce throughput by 50%+ under concurrent load
Fix Strategy: Padding shared memory arrays or stride pattern changes

Context Switching Resource Limits

Symptom: Individual kernels work, crashes when run together
Root Cause: Multiple kernels exceeding GPU resource limits
Detection: nvidia-smi -q -d UTILIZATION during concurrent execution
Solution: Reduce concurrent launches or increase memory bandwidth with CUDA streams

Critical Warnings - What Documentation Doesn't Tell You

CUDA Graphs Debugging Limitations

• Even CUDA_LAUNCH_BLOCKING=1 only catches graph-level failures
• Core dumps still trigger inside graphs (only reliable debugging method)
• Split kernels out of graphs temporarily for debugging

Multi-GPU Coordination Under Load

• Race conditions only appear with realistic concurrency levels
• NCCL_DEBUG=INFO essential for tracing collective operations
• Uneven workload distribution causes indefinite waits on synchronization

Container GPU Debugging Requirements

# Minimum container setup for debugging
docker run --gpus all --cap-add=SYS_PTRACE your_image

# For compute-sanitizer (security risk - debug only)
docker run --gpus all --privileged your_image

Framework Memory Pool Deception

• nvidia-smi shows allocated memory, not reserved memory
• PyTorch caching allocator hides true memory usage patterns
• Always use framework-specific memory monitoring tools

Decision Criteria for Debugging Approaches

When to Use Each Tool

• Core dumps: Illegal memory access with wrong stacktraces
• compute-sanitizer: Suspected race conditions or memory corruption
• cuda-gdb: Need actual GPU stack traces (when it works)
• Nsight Compute: Performance issues or memory access pattern problems

Hardware vs Software Problem Indicators

Hardware Problems:

• ECC errors (any double-bit errors = hardware failure)
• Progressive performance degradation (thermal throttling)
• Random crashes only under load (memory errors)

Software Problems:

• Consistent crash patterns
• Wrong results with deterministic inputs
• Memory leaks with predictable growth

Production Debugging Priorities

1. Check hardware health (temperature, ECC errors, power delivery)
2. Verify framework-specific memory usage patterns
3. Test with realistic concurrency levels
4. Confirm driver/kernel version compatibility
5. Use synchronous execution and core dumps for error location

Breaking Points and Failure Modes

Critical GPU Memory Thresholds

• Fragmentation: Available memory <80% of free memory indicates fragmentation
• Bank conflicts: >12% conflict rate causes visible performance degradation
• Thermal throttling: >87°C triggers automatic frequency reduction
• ECC threshold: Any double-bit errors require immediate hardware replacement

Driver Update Risk Assessment

• Ubuntu unattended upgrades: High risk of nouveau driver conflicts
• Kernel version changes: Require NVIDIA driver reinstallation
• CUDA toolkit mismatches: nvidia-smi vs nvcc --version discrepancies

Multi-GPU Scaling Limitations

• NCCL deadlock probability: Increases exponentially with GPU count
• Memory bandwidth saturation: >4 GPUs often limited by PCIe bandwidth
• Synchronization overhead: Collective operations become bottleneck at scale

This knowledge base provides structured, actionable intelligence for debugging CUDA production issues, focusing on failure patterns, tool limitations, and hardware-level thinking required for effective GPU application debugging.