NVIDIA Triton Performance Tuning: Production-Ready Technical Reference

Critical Configuration Settings

Dynamic Batching Production Configuration

dynamic_batching {
  max_queue_delay_microseconds: 50000
  preferred_batch_size: [4, 8]
  max_queue_size: 256
}

Why These Values Work:

• max_queue_delay_microseconds: 50000 (50ms): Maximum wait time before users perceive API as broken
• preferred_batch_size: [4, 8]: Sweet spot for transformers - smaller batches start immediately, larger improve throughput
• max_queue_size: 256: Prevents memory exhaustion during traffic spikes

TensorRT Optimization Configuration

optimization { execution_accelerators {
  gpu_execution_accelerator : [ {
    name : "tensorrt"
    parameters { key: "precision_mode" value: "FP16" }
    parameters { key: "max_workspace_size_bytes" value: "2147483648" }
    parameters { key: "trt_engine_cache_enable" value: "1" }
  }]
}}

Model Instance Configuration

instance_group [
  { count: 2, kind: KIND_GPU, gpus: [0] }
]

Critical Limit: Maximum 2 instances - more causes memory overhead and scheduler load balancing bugs.

Performance Baselines (ResNet-50 on A100)

Configuration	Throughput (infs/sec)	P95 Latency (ms)	Real-World Impact
Baseline (no optimization)	380-420	28	Starting point
Dynamic batching only	1150-1300	42-48	100-150% improvement
Dynamic batching + 2 instances	1650-1850	52-58	Diminishing returns visible
Full optimization + TensorRT	2200-2500	33-38	Maximum achievable

Testing Methodology: 16 concurrent clients, 10-minute runs minimum (shorter tests produce unreliable data).

Critical Failure Modes and Solutions

Memory Leak Crisis

Problem: Dynamic batching in recent Triton versions has memory leaks where batched requests aren't garbage collected
Symptoms: Memory usage climbs continuously until OOM crash after 6-8 hours
Immediate Fix:

0 */4 * * * docker restart triton-server

Root Cause: GitHub issue #6854 - known bug affecting production deployments

TensorRT Compilation Delays

Problem: First model load triggers 15-30 minute compilation phase
Impact: Health checks timeout, deployment failures
Solutions:

1. Pre-compile engines outside Triton:

trtexec --onnx=model.onnx --saveEngine=model.engine --fp16

2. Use pre-compiled engines with platform: "tensorrt_plan"

Scheduler Load Balancing Failures

Problem: Round-robin scheduling bug sends all requests to instance 0
Symptoms: Uneven load distribution visible in nvidia-smi
Workaround: Limit to 2 instances maximum, monitor with GPU utilization tools

Resource Requirements and Constraints

Memory Overhead Calculations

• TensorRT engines: 500MB ONNX model → 2GB TensorRT engine (4x increase)
• Multiple instances: Non-linear memory usage due to CUDA context overhead
• Dynamic batching: Queue memory scales with max_queue_size

TensorRT Compatibility Matrix

Models That Work (60% success rate):

• ResNet variants
• EfficientNet
• Basic CNNs
• Simple transformers

Models That Fail:

• Dynamic shapes (unsupported)
• Custom ONNX operators
• Newer model architectures

Time Investment Requirements

• TensorRT optimization: 15-30 minutes per model compilation
• Production tuning: 6+ months to achieve stable configuration
• Debugging cycles: 24-hour testing minimum per configuration change

Decision Criteria and Trade-offs

Dynamic Batching vs Memory Stability

Choose Dynamic Batching When:

• Can implement 4-hour restart schedule
• Memory monitoring infrastructure exists
• 100-150% performance gain justifies operational overhead

Avoid When:

• High availability requirements (>99.9% uptime)
• Limited monitoring capabilities
• Memory-constrained environments

TensorRT vs Operational Simplicity

Use TensorRT When:

• Model confirmed compatible (test with trtexec first)
• 15-30 minute compile times acceptable
• 2x performance gain essential

Avoid When:

• Dynamic input shapes required
• Rapid iteration/deployment cycles
• Development environments (compilation overhead too high)

Production Debugging Procedures

Memory Issue Diagnosis

1. GPU Memory Monitoring:

nvidia-smi dmon -s u -i 0

2. Queue Depth Analysis:
   • Enable Triton metrics endpoint
   • Monitor nv_inference_queue_duration for spikes
3. Batch Size Verification:
   • Log actual vs expected batch sizes
   • Check for queue backing up

Performance Degradation Investigation

Primary Indicators:

• GPU utilization <80% with high latency
• Memory usage climbing over time
• Queue duration spikes

Root Cause Analysis:

1. Check scheduler load distribution across instances
2. Verify batch size configuration matches workload
3. Analyze memory leak patterns over 24-hour periods

Tool Reliability Assessment

Performance Analysis Tools

Tool	Reliability	Setup Complexity	Use Case
perf_analyzer	High (recommended)	Low	Production benchmarking
Model Analyzer	Low (frequent crashes)	High	Avoid - use perf_analyzer
GenAI-Perf	Medium	High	LLM-specific testing only
nvidia-smi	High	None	GPU monitoring

Essential Command Examples

# Throughput testing
perf_analyzer -m model_name -b 8 --concurrency-range 1:32:4 --measurement-interval 60000

# Latency testing
perf_analyzer -m model_name --latency-threshold 100000 --measurement-mode count_windows

# Memory monitoring
nvidia-smi -l 1

Production Deployment Checklist

Pre-Deployment Validation

• TensorRT compatibility tested with trtexec
• Memory requirements calculated (4x overhead for TensorRT)
• Batch size configuration validated with realistic load
• Instance count limited to 2 maximum
• Restart schedule implemented for memory leak mitigation

Monitoring Requirements

• GPU memory utilization tracking
• Queue depth metrics enabled
• Batch size logging implemented
• 24-hour performance baseline established

Failure Recovery Procedures

• Automatic restart every 4 hours (memory leak mitigation)
• Fallback to ONNX model if TensorRT fails
• Instance count reduction procedure for memory pressure
• Emergency batch size reduction configuration

Hidden Costs and Expertise Requirements

Human Resource Investment

• Initial Setup: 2-4 weeks for experienced ML engineers
• Production Tuning: 3-6 months to achieve stability
• Ongoing Maintenance: 1 day/month for monitoring and updates

Infrastructure Overhead

• Memory: 300-400% increase for TensorRT optimization
• Compute: 15-30 minutes compilation time per model update
• Monitoring: Full GPU utilization and queue depth tracking required

Community Support Quality

• Triton GitHub Issues: High responsiveness, NVIDIA engineers participate
• TensorRT Issues: Better than average open source support
• NVIDIA Forums: Mixed quality, check dates on advice (often outdated)

This technical reference provides actionable configuration guidance while preserving critical operational intelligence about failure modes, resource requirements, and real-world performance expectations.