Clair Production Monitoring - Keep Your Scanner Running (Or Watch Everything Break)

PostgreSQL: Where Performance Goes to Die

Your PostgreSQL database is the heart of Clair, and it's probably your biggest operational headache. When teams first deploy Clair, they slap it on a basic RDS instance and assume it'll scale. Wrong.

Connection pool exhaustion hits first. The default config allows 100 connections, but three indexer instances can easily saturate that during peak scanning. You'll see connection pool exhausted errors in your logs, and new scans just hang. PostgreSQL connection pooling becomes critical - bump max_connections to 200+ and configure PgBouncer if you're serious about scale.

Database bloat kills query performance once you hit 500,000+ indexed images. The vulnerability correlation queries that worked fine with 10,000 images take 30+ seconds with real production data. VACUUM operations need to run regularly, especially after vulnerability database updates that touch millions of rows.

Memory consumption spirals during updater runs in ways you can't predict. When Ubuntu releases daily security updates, the RHEL VEX updater (new in v4.8.0) rebuilds correlation data that might use 8GB of RAM or might use 16GB - honestly, I think it depends on the phase of the moon. Your database server needs at least 16GB to handle these spikes, but I've personally watched it blow past that and there's no good monitoring for this shit.

Memory Usage: The Silent Container Killer

Clair's memory consumption is unpredictable and absolutely brutal. A basic Ubuntu container might index with 200MB RAM usage, but that TensorFlow container with 73 layers and custom Python packages? I've seen it spike anywhere from 4-6GB to "oh fuck it's eaten our entire node's memory" levels during indexing. There's literally no way to predict this shit ahead of time.

Kubernetes resource limits become a nightmare to tune. Set them too low and your indexer pods get OOMKilled mid-scan. Set them too high and you're wasting cluster resources. I've seen production deployments where 80% of memory allocation goes unused most of the time, but you need those spikes handled.

The worst part is memory leaks during malformed container analysis. When Clair encounters a corrupted layer or some weird-ass package metadata that shouldn't exist, memory usage climbs indefinitely until the process crashes. I've personally debugged situations where you just see the pod restart at 3am, lose 45 minutes of scan progress, and have no fucking idea why because there's no decent monitoring for this.

Webhook Delivery: The Async Disaster

Webhook notifications fail silently and spectacularly. Your vulnerability notification system depends on these webhooks, but when they break, you might not notice for weeks.

Timeout configurations cause most webhook failures. The default 30-second timeout seems generous until you realize webhook processing might include Slack notifications, database updates, and policy evaluations. Your receiving endpoint times out, Clair marks it as failed, and you stop getting vulnerability alerts.

Webhook retry logic is primitive. Failed deliveries get retried with exponential backoff, but there's no dead letter queue or manual retry mechanism. If your webhook endpoint is down for an hour, you lose all notifications from that window.

Authentication failures happen constantly with token rotation. Your webhook endpoint expects a valid JWT, but when certificates rotate or tokens expire, webhook delivery just stops. Clair logs show authentication failed but doesn't distinguish between temporary and permanent auth failures.

Vulnerability Database Updates: The Scanning Killer

Database updates lock scanning operations and create unpredictable downtime. When RHEL VEX data updates (the new hotness in v4.8.0), the matchers rebuild correlation indexes that block all vulnerability queries.

The migration from RHEL OVAL to VEX updater in Clair v4.8.0 creates a specific operational nightmare. During the upgrade, there's a period where no Red Hat vulnerabilities exist in the database until the VEX updater completes its first run. Production deployments can go hours without RHEL vulnerability detection.

Network dependency failures cascade across the entire system. Clair needs to fetch updates from NVD, Ubuntu USN, Debian DSA, and now Red Hat VEX endpoints. When any of these services are slow or unavailable, updater runs hang and block the entire scanning pipeline.

Rate limiting from upstream sources breaks update schedules. The NVD API started enforcing rate limits that can delay vulnerability database synchronization by hours. Your scanning pipeline looks healthy, but you're working with day-old vulnerability data.

Metrics That Matter (And Ones That Don't)

Most teams monitor the wrong shit. CPU and memory utilization tell you nothing useful about Clair's operational health. I've watched a Clair instance show 20% CPU usage while the scan queue backs up for hours because PostgreSQL connection pools are fucked.

Clair's Prometheus metrics include clair_indexer_queue_size, but this only shows queued scan requests, not the time those requests have been waiting. You need custom alerting that tracks scan completion times: if a basic Ubuntu container takes more than 2 minutes to index, something's wrong.

Database metrics reveal more operational problems than application metrics. Monitor PostgreSQL `pg_stat_activity` to track connection usage and query duration. When connection counts hit 80% of your limit, you're about to hit the operational cliff where new scans hang indefinitely.

Vulnerability database update success is binary but critical. `clair_updater_last_success` timestamps show when each updater completed successfully. If your RHEL VEX updater (new in v4.8.0) hasn't succeeded in 24+ hours, you're missing critical security data.

Log Analysis for Operational Intelligence

Clair's logs actually tell you what's broken if you know how to read them. Connection pool exhaustion shows up as acquiring connection: timeout before scan failures become visible to users - I learned to alert on this pattern after spending a weekend debugging stuck scans.

Memory allocation failures manifest as runtime: out of memory in indexer logs, but by then it's too late. Monitor for increasing memory usage patterns during specific container types - ML containers with 50+ layers almost always trigger memory spikes.

Webhook delivery failures appear as notification delivery failed with HTTP status codes. Status 5xx errors indicate temporary problems worth retrying; 4xx errors usually mean authentication or configuration problems that need manual intervention.

Database query performance degradation shows up as slow query warnings, but Clair's default log level misses these. Enable query logging in PostgreSQL with log_min_duration_statement = 1000 to catch queries taking longer than 1 second.

Alerting Rules That Don't Create Noise

Critical alerts (page someone immediately):

• PostgreSQL connection pool above 90% utilization for 2+ minutes
• Any indexer pod OOMKilled in the last 5 minutes
• Vulnerability database updaters failing for 6+ hours
• Scan queue size above 100 requests for 10+ minutes

Warning alerts (Slack notification):

• Individual container scans taking 5+ minutes consistently
• Memory usage above 3GB for basic container indexing
• Webhook delivery failure rate above 10% over 1 hour
• Database query times above 5 seconds average

Informational tracking (metrics only):

• Daily scan volume and completion times
• Vulnerability database update frequency and duration
• Resource usage patterns by container type and size

What to Expect When Everything Goes Wrong

Look, you need to know what normal performance looks like before you can tell when shit's broken. A standard Ubuntu base image should index in under 30 seconds. Multi-stage Docker builds with 20+ layers typically take 1-2 minutes. ML containers with custom compiled packages can legitimately take 5+ minutes - and that's not necessarily your problem.

Database growth follows predictable patterns until it doesn't. Every 1,000 indexed containers generates roughly 500MB of PostgreSQL data. Plan for 50GB+ database storage if you're scanning 100,000+ images regularly - the vulnerability and vuln_affected tables are where all your disk space disappears.

Memory usage makes no fucking sense. A 4GB container with simple package metadata might use 500MB during indexing, while a 500MB container with complex Python environments can spike to 6GB. Size alone tells you nothing about what resources you'll need.

Network bandwidth needs scale with container layers and registry distance. Local registries enable 100+ Mbps sustained transfer during indexing. External registries (Docker Hub, ECR across regions) might limit you to 10-20 Mbps, significantly impacting scan throughput.

Health Check Strategies Beyond HTTP 200

HTTP health checks miss most operational problems. Clair's /healthz endpoint returns 200 even when the scan queue is backed up for hours or webhook delivery is failing.

Database connectivity health checks should verify both connection availability and query performance. A simple SELECT 1 proves connectivity but misses performance degradation. Use SELECT COUNT(*) FROM vulnerability LIMIT 1 to test query responsiveness.

Functional health checks should attempt actual scanning operations. Submit a small, known container for indexing and verify completion within expected time bounds. This catches integration problems that infrastructure health checks miss.

End-to-end monitoring should track the complete vulnerability reporting pipeline. Submit a container with known vulnerabilities, verify scan completion, and confirm webhook delivery to your notification system. This proves the entire system works, not just individual components.