Trivy Container Scanning: Production Troubleshooting Guide

Critical Configuration Requirements

Memory Resource Allocation

Production Minimums:

• Alpine Linux containers: 2GB RAM
• Node.js/Python applications: 4GB RAM
• Java/Spring Boot applications: 8GB RAM
• ML frameworks (TensorFlow/PyTorch): 16GB+ RAM

Failure Pattern: Memory consumption spikes during JAR analysis phase, not gradually. Container sits at 512MB for 10 minutes, then instantly consumes 6GB during dependency analysis.

Exit Code 137 Solution:

docker run --memory=8g --memory-swap=16g aquasec/trivy:latest image your-spring-boot-app:latest

Database Download Configuration

GitHub API Rate Limits:

• Unauthenticated: 60 requests/hour
• Authenticated: 5,000 requests/hour

Required Authentication:

export GITHUB_TOKEN="your-personal-access-token"
trivy image --timeout 20m your-image:latest

Pre-download Strategy (85% success rate):

# Separate database download from scanning
trivy image --download-db-only
trivy image --download-java-db-only
trivy image --skip-db-update your-image:latest

Failure Modes and Solutions

Memory Exhaustion (OOMKilled/Exit Code 137)

Success Rate: 90% with memory increase

• Symptom: Container killed during JAR analysis phase
• Root Cause: Resource limits exceeded during dependency tree analysis
• Solution: Increase container memory limits or use server mode

Database Download Timeouts

Success Rate: 85% with pre-download strategy

• Symptom: Hangs on "Downloading vulnerability DB" for 20+ minutes
• Root Cause: Network timeouts, proxy interference, or API rate limiting
• Solution: GitHub token authentication + pre-download databases

Network/Proxy Issues

Corporate Environment Failures:

• SSL inspection breaks signature verification
• VPN packet loss corrupts database cache
• Proxy timeouts during database downloads

Configuration Fix:

export HTTP_PROXY="http://proxy.company.com:8080"
export HTTPS_PROXY="http://proxy.company.com:8080"
export NO_PROXY="localhost,127.0.0.1,.company.com"

Performance Characteristics

Scan Duration by Image Type

• Alpine Linux: 30 seconds
• Node.js applications: 2-5 minutes
• Spring Boot applications: 10-15 minutes
• TensorFlow/ML images: 30+ minutes (high failure risk)

Resource Usage Patterns

Memory consumption follows predictable pattern:

1. Low usage during setup (512MB)
2. Massive spikes during JAR analysis (6-8GB)
3. Gradual decline during vulnerability matching

Production Architecture Solutions

Server Mode (95% Success Rate)

Separates resource-intensive scanning from client environment:

# Run dedicated scanning server
trivy server --listen 0.0.0.0:4954 --cache-dir /tmp/trivy-cache

# Client scanning with resource isolation
trivy image --server YOUR_TRIVY_SERVER:4954 your-image:latest

CI/CD Integration Requirements

Dedicated Scanning Infrastructure:

• Separate scanning nodes from build infrastructure
• Minimum t3.medium instances (4GB RAM)
• Shared cache volumes for database reuse
• Proper timeout configuration (20m+ for complex images)

Resource Planning Multipliers:

• Base workload + 3-5x multiplier for complex images
• Peak memory during dependency analysis phases
• Account for scanning spikes during CI/CD peak hours

Critical Warnings

What Will Break in Production

1. t2.micro instances - Insufficient memory for real applications
2. Shared CI/CD resources - Resource contention causes failures
3. Default timeouts - Inadequate for complex dependency analysis
4. Missing GitHub tokens - API rate limiting guaranteed failure
5. Corporate proxies without SSL bypass - Database corruption

Hidden Costs

• Time Investment: 3+ months to resolve enterprise network issues
• Expertise Requirements: DevOps + security team coordination
• Infrastructure Costs: Dedicated scanning instances required
• Maintenance Overhead: Daily database refresh automation needed

Alternative Solutions

When Trivy Fails

Fallback Scanning Tools:

• Grype: Lower memory usage, faster scanning
• Snyk: Better Docker Desktop integration
• Anchore: Enterprise policy management

Decision Criteria:

• Memory constraints → Grype
• Enterprise policies → Anchore
• Developer workflow → Snyk
• Air-gapped environments → Custom database management

Monitoring and Prevention

Success Metrics

• Scan completion rate: >95%
• Zero OOMKilled failures per week
• Database download success: >99%
• Average scan time within 2x baseline

Infrastructure Monitoring

# Resource tracking during scans
docker stats --no-stream trivy_container_id

# Failure pattern logging
trivy image --timeout 30m your-image:latest 2>&1 | tee scan-$(date +%Y%m%d).log

Alert Thresholds

• Memory usage exceeding 80% during scans
• Scan duration exceeding baseline by 200%
• Persistent database download failures
• Network timeout patterns

Enterprise Implementation Strategy

Phase 1: Infrastructure Preparation

1. Provision dedicated scanning instances (t3.large minimum)
2. Configure GitHub tokens and proxy settings
3. Implement shared cache volumes
4. Set up database pre-download automation

Phase 2: CI/CD Integration

1. Separate scanning from build pipelines
2. Implement proper error handling for exit codes
3. Configure resource monitoring and alerting
4. Establish fallback scanning options

Phase 3: Optimization

1. Image layer optimization for faster scanning
2. Dependency caching strategies
3. Performance baseline establishment
4. Automated scaling based on scan volume

Resource Requirements Summary

Minimum Viable Production Setup

• Compute: t3.medium (4GB RAM) minimum
• Storage: 50GB for database cache
• Network: Direct internet or properly configured proxy
• Authentication: GitHub personal access token
• Monitoring: Resource usage tracking and alerting

Cost-Performance Trade-offs

• Bigger instances: Higher cost, higher reliability
• Server mode: Infrastructure complexity, better resource isolation
• Pre-download databases: Network overhead, scanning reliability
• Alternative scanners: Tool learning curve, different vulnerability coverage