Clair Production Monitoring: AI-Optimized Knowledge

Critical Failure Scenarios & Production Breaking Points

PostgreSQL Database Failures

Breaking Point: 100 connections (default limit) - three indexer instances saturate connection pool
Impact: New scans hang indefinitely, vulnerability reports stop generating
Resource Requirements: Minimum 200+ connections, 16GB RAM, PgBouncer for connection pooling
Hidden Cost: Database bloat after 500,000+ indexed images causes 30+ second query times

Memory Consumption Spikes

Unpredictable Range: 200MB (basic Ubuntu) to 8GB+ (TensorFlow containers with custom packages)
Critical Failure: Memory leaks during malformed container analysis - process crashes at 3am
Kubernetes Impact: Setting limits too low = OOMKilled pods, too high = 80% wasted allocation
No Prediction Method: Container size doesn't correlate with memory usage

Webhook Delivery Silent Failures

Default Timeout: 30 seconds (insufficient for complex processing chains)
Authentication Failure Mode: Token rotation breaks delivery without alerts
Retry Logic Limitation: No dead letter queue, exponential backoff only
Security Impact: Missing vulnerability notifications for weeks

Vulnerability Database Update Locks

Scan Blocking: RHEL VEX updates (v4.8.0+) lock all vulnerability queries
Migration Gap: 2-6 hours without Red Hat vulnerability detection during v4.8.0 upgrade
Network Dependencies: NVD, Ubuntu USN, Debian DSA, Red Hat VEX - any failure cascades
Rate Limiting Impact: NVD delays can push updates 6+ hours behind

Configuration That Actually Works in Production

PostgreSQL Settings

max_connections = 200+
autovacuum_max_workers = 6+
autovacuum_vacuum_scale_factor = 0.1
log_min_duration_statement = 1000ms

Memory Allocation Strategy

• Base Containers: 1GB limit minimum
• ML/Complex Containers: 8GB+ limit required
• Database Server: 16GB minimum for VEX update spikes

Required Indexes for Scale

CREATE INDEX CONCURRENTLY idx_vuln_affected_package 
ON vuln_affected (package_id, vulnerability_id);

Monitoring Thresholds

• Critical Alert: PostgreSQL connections >90% for 2+ minutes
• Critical Alert: Scan queue >100 requests for 10+ minutes
• Warning: Individual scans >5 minutes consistently
• Warning: Memory usage >3GB for basic containers

Operational Intelligence & Troubleshooting

Performance Baselines

• Standard Ubuntu: <30 seconds indexing
• Multi-stage builds (20+ layers): 1-2 minutes
• ML containers: 5+ minutes (legitimate)
• Database growth: 500MB per 1,000 containers
• Storage planning: 50GB+ for 100,000+ images

Common Failure Patterns

1. Connection pool exhaustion (70% of stuck scans) - Check pg_stat_activity
2. Network timeouts - Look for context deadline exceeded in logs
3. Silent OOMKills - Check kubectl describe pod for kill events
4. Query performance cliff - Happens around 100,000 indexed images

Migration Risks (v4.8.0 OVAL to VEX)

Procedure: Run clairctl -D admin pre v4.8.0 during maintenance window
Downtime: 2-6 hours of missing Red Hat vulnerability detection
Failure Mode: Authentication issues with new VEX endpoints
Rollback Complexity: High - test migration procedures in staging first

Resource Requirements & Scaling

Unpredictable Factors

• Container complexity (not size) determines memory usage
• ML containers with Python packages = highest resource consumption
• Binary-only containers = minimal resources regardless of size
• Network distance to registry = 10x performance impact

Infrastructure Dependencies

• Local registries (Harbor): 100+ Mbps sustained transfer
• External registries (Docker Hub): 10-20 Mbps limitation
• Air-gapped environments: Require vulnerability database mirroring

Critical Warnings & Known Issues

Silent Failure Modes

• HTTP health checks miss operational problems (/healthz returns 200 during queue backup)
• Webhook delivery failures don't distinguish temporary vs permanent failures
• Memory leaks with malformed containers have no reliable detection

Vendor-Specific Issues

• AWS ECR: IAM role issues cause intermittent authentication failures
• Docker Hub: Rate limiting blocks anonymous layer downloads
• Harbor: Provides best operational experience with built-in integration

Air-Gapped Deployment Complexity

• Certificate validation fails with custom CA hierarchies
• Database migration procedures need isolation testing
• Vulnerability feed synchronization requires custom scripting

Decision Support Matrix

When to Use Clair

Worth it despite complexity: Large container inventories requiring compliance reporting
Not recommended: Small deployments (<1000 containers) due to operational overhead
Alternative consideration: Harbor integration reduces operational burden significantly

Resource Investment Requirements

• Time: 2-4 weeks for production-ready deployment
• Expertise: PostgreSQL DBA knowledge essential
• Monitoring: Custom Prometheus/Grafana setup required
• Maintenance: Weekly database VACUUM operations

Breaking Change Impact

• Version upgrades: Require database migrations with rollback testing
• Dependency changes: Upstream rate limiting affects update schedules
• Security patches: May require extended maintenance windows

Monitoring Implementation Guide

Essential Metrics (Not CPU/Memory)

• clair_indexer_queue_size - Alert when >100
• clair_updater_last_success - Alert when >24 hours old
• PostgreSQL connection count - Alert at 80% utilization
• Scan completion time tracking - Alert when >2x baseline

Log Analysis Patterns

• acquiring connection: timeout = Connection pool exhaustion
• runtime: out of memory = Memory allocation failure (too late to act)
• notification delivery failed = Webhook issues with HTTP status codes
• slow query warnings = Query performance degradation

Functional Health Checks

Submit known container for indexing and verify:

1. Completion within expected time
2. Vulnerability detection accuracy
3. Webhook delivery to notification system
4. Database query responsiveness

This operational intelligence enables AI systems to make informed decisions about Clair deployment, resource allocation, and troubleshooting procedures based on real-world production experience.