HashiCorp Vault + Kubernetes CI/CD: Production Implementation Guide

Configuration Requirements

Dynamic Database Credentials

• Vault creates actual database users with UUID suffixes and precise permissions
• TTL expires automatically - credentials stop working without manual cleanup
• Supported databases: PostgreSQL, MySQL, MongoDB, AWS RDS, dozens more
• Performance impact: Each secret request creates/revokes database users
• Connection limits critical: Postgres requires 200+ max_connections (default 100 fails)
• Real failure mode: 100 pods requesting secrets = database connection exhaustion at 95% CPU

Authentication Methods (Production Reality)

Method	RAM Usage	Failure Mode	Recovery
Vault Agent Injector	~100MB per pod	Init:0/1 with useless logs	Delete pod, recreate
External Secrets Operator	No pod overhead	Stops syncing silently for weeks	Restart operator
Direct API Calls	Minimal overhead	OIDC tokens expire mid-deployment	Split long jobs
CSI Driver	Low overhead	Mount failures kill pods	Restart affected pods

Network Policy Requirements

Critical connections to allow:

1. Agent Injector → Vault (port 8200)
2. Vault → Kubernetes API (port 6443 for token validation)
3. Pods → Vault (if using direct API)
4. ESO → Vault (if using External Secrets)

Common failure: Cilium 1.14.x randomly blocks cross-namespace calls with context deadline exceeded

Resource Requirements

Time Investments

• Initial setup: 2-3 days for basic implementation
• Network policy debugging: Full day when enabling security policies
• Production hardening: 1-2 weeks including monitoring and disaster recovery
• Authentication troubleshooting: 8+ hours for complex RBAC issues

Expertise Requirements

• Kubernetes RBAC: Essential for service account configuration
• Database administration: Understanding connection pooling and user management
• Network debugging: Required for policy troubleshooting
• OIDC/JWT knowledge: Critical for CI/CD pipeline authentication

Infrastructure Costs

• Vault cluster: 3+ nodes for HA, Enterprise for disaster recovery
• Database connections: 2-3x normal connection limits
• Monitoring: Prometheus, Grafana, audit log storage
• Backup storage: For Vault snapshots and disaster recovery

Critical Warnings

Token Expiration Issues

• GitHub Actions OIDC tokens: 5-minute lifespan, pipelines fail at final deploy
• Kubernetes service account tokens: 1-hour default in new clusters
• GitLab CI tokens: 10-minute maximum before becoming worthless
• Real impact: Entire Friday release pipeline died during Trivy security scan

External Secrets Operator Failures

• Silent failures: Shows "successfully refreshed" while secrets expire
• Duration: Can stop working for 3+ weeks without alerts
• Impact: 47+ services failing simultaneously at 3:15 AM
• Root cause: ESO lies in logs while service account tokens expire

Database Performance Degradation

• Connection exhaustion: FATAL: too many connections for role "vault"
• Concurrent user creation: Postgres 13 struggles, 14+ handles better
• Rolling deployments: 100 pods requesting secrets overwhelms database
• Recovery time: Weeks of connection limit alerts after fixes

Vault Downtime Consequences

• Complete CI/CD failure: All pipelines stop when Vault unavailable
• No automatic failover: Requires manual intervention or Enterprise features
• Network policy conflicts: Mysterious timeouts with zero useful errors
• Recovery complexity: Unsealing, token validation, service restoration

Implementation Strategies

Production-Ready Configuration

# Vault Agent memory limits (per pod overhead)
resources:
  limits:
    memory: "128Mi"  # Actual usage ~100MB
    cpu: "100m"
  requests:
    memory: "64Mi"
    cpu: "50m"

# Database connection configuration
max_connections: 200  # Minimum for production clusters
connection_timeout: "30s"
lease_duration: "4h"  # Balance between security and operational overhead

Monitoring Requirements

Essential alerts (prevent 3AM pages):

• Secret request latency > 5 seconds
• Authentication failure rate > 5%
• Token expiration warnings (10 minutes before)
• Vault unsealed status
• Database connection count approaching limits

Useful metrics:

• Secret request error rate
• Database user creation/deletion patterns
• Network policy blocking events
• OIDC token refresh failures

Failure Recovery Procedures

Agent Injector stuck in Init:0/1:

1. Check service account has system:auth-delegator cluster role
2. Verify Agent Injector webhook can reach Kubernetes API
3. Confirm network policies allow Vault connectivity
4. Nuclear option: Delete pod and recreate (40% success rate)

External Secrets stopped syncing:

1. Restart ESO operator pods (fixes 90% of cases)
2. Check Vault token expiration
3. Verify network policy ingress rules
4. Force refresh with shorter TTL values

CI/CD authentication failures:

1. Verify OIDC token availability in pipeline
2. Check Vault OIDC configuration (GitHub changed URLs in March 2024)
3. Split long deployments into shorter jobs
4. Implement caching between pipeline stages

Decision Criteria

When to Use Vault + Kubernetes

Worth the complexity if:

• Storing secrets in Git is unacceptable
• Compliance requires audit trails (SOC 2, HIPAA, FedRAMP)
• Dynamic credentials reduce rotation overhead
• Multiple environments need secret isolation

Not worth it if:

• Simple applications with few secrets
• Team lacks Kubernetes/Vault expertise
• Can't dedicate resources to ongoing maintenance
• Vault downtime would be catastrophic

Integration Method Selection

Use Agent Injector when:

• Applications can't be modified for API calls
• Memory overhead acceptable (~100MB per pod)
• Team comfortable debugging init container failures

Use External Secrets when:

• GitOps workflow required
• Standard Kubernetes Secret objects preferred
• Can tolerate sync delays and silent failures

Use Direct API when:

• Maximum performance required
• Team has OIDC/JWT expertise
• Can handle token expiration complexity

Alternative Solutions

Consider instead:

• Sealed Secrets: Simpler for Git-based workflows
• AWS Secrets Manager: If already on AWS
• Azure Key Vault: If already on Azure
• Google Secret Manager: If already on GCP
• SOPS: For encrypted secrets in Git

Breaking Points and Failure Modes

Scale Limitations

• Database user creation: Rate-limited by database performance
• Memory usage: Agent Injector scales linearly with pod count
• Network connections: Vault clustering required beyond single cluster
• Audit log volume: Can overwhelm logging infrastructure

Security Considerations

• Cluster-admin requirements: Initial setup needs elevated privileges
• Network exposure: Vault API accessible within cluster
• Secret leakage: Temporary files on disk during injection
• Audit gaps: Silent failures don't generate audit events

Operational Overhead

• 24/7 monitoring required: Vault outages impact all services
• Expertise requirements: Complex troubleshooting scenarios
• Disaster recovery complexity: Enterprise features for full automation
• Version compatibility: Kubernetes/Vault version matrix management

Production Success Indicators

• Secret request latency consistently < 2 seconds
• Authentication failure rate < 1%
• Zero silent External Secrets sync failures
• Database connection utilization < 70%
• Vault cluster availability > 99.9%
• Complete audit trail for compliance requirements
• Automated disaster recovery tested quarterly