HashiCorp Vault + Kubernetes: Stop Committing Database Passwords to Git

Look, managing secrets in Kubernetes sucks. You've probably been through this dance before: hardcode some database passwords in your deployment YAML "just for testing," forget to rotate them for six months, then scramble when security scans your Git history and finds plaintext AWS keys from 2019.

Why Your Current Secret Management Is Broken

Environment Variable Hell: You're storing secrets in ConfigMaps or worse, directly in your deployment specs. Every time you need to rotate a password, you're redeploying pods and crossing your fingers nothing breaks. Kubernetes secrets aren't much better - they're just base64 encoded plaintext stored in etcd.

The Manual Rotation Nightmare: Remember that database password that expired last weekend and took down production? Yeah, that's what happens when rotation is a manual process that relies on Larry remembering to update the secret before he goes on vacation. Security guides suggest automated 30-90 day cycles, but who has time for that?

Permission Sprawl: Your CI/CD service account probably has access to every secret in every namespace because nobody wanted to spend three hours debugging RBAC permissions. One compromised pipeline = game over. Principle of least privilege? More like principle of "fuck it, just give it cluster-admin."

Vault's Approach (When It Actually Works)

HashiCorp Vault generates secrets when you need them and kills them automatically. Instead of storing DB_PASSWORD=hunter123 in your environment variables, your app asks Vault for a fresh database user that expires in 4 hours.

Dynamic Database Credentials: Vault creates actual database users with precise permissions for each deployment. When the TTL expires, those credentials stop working. No manual cleanup required. Works with PostgreSQL, MySQL, MongoDB, and dozens of other databases.

Short-Lived Everything: API keys, certificates, cloud tokens - all generated with TTLs measured in hours, not months. Compromised credential? Wait it out or revoke immediately through Vault's API. AWS STS tokens can be generated on-demand with 15-minute lifespans.

Audit Logs That Actually Help: Every secret request includes who asked, when, from where, and what they got. Perfect for those fun compliance audits where you need to prove you're not storing passwords in Git. SOC 2, HIPAA, FedRAMP - all covered with proper audit device configuration.

Three Ways to Get Secrets From Vault (All With Trade-offs)

Option 1: Vault Agent Injector - Works great when it works. When it doesn't, good luck figuring out why your pod is stuck in Init:0/1 forever. The Agent Injector error messages are about as helpful as a chocolate teapot. Check the common issues guide when shit hits the fan.

Option 2: External Secrets Operator - Probably your best bet unless you have exotic requirements. ESO syncs Vault secrets into regular Kubernetes Secret objects. Sometimes stops syncing secrets for mysterious reasons, but at least you can kubectl your way to a solution. The ESO documentation is actually readable, unlike most Kubernetes projects.

Option 3: Direct API calls - For masochists who enjoy debugging OAuth flows at 3am. Your GitHub Actions or GitLab CI jobs authenticate directly to Vault using OIDC tokens. Fast when it works, nightmare when GitHub's OIDC provider has a bad day. JWT authentication is another option but equally painful to debug.

The Real Problems Nobody Talks About

OIDC Token Expiration: GitHub Actions OIDC tokens last exactly 5 minutes. Your deployment takes 20 minutes. I found this out when our entire Friday release pipeline died at the final Helm deploy step with Error: invalid token audience "vault://vault.example.com" because the token expired during our Trivy security scan. GitLab CI tokens pull the same bullshit - good for 10 minutes max before they turn into worthless JWT garbage.

Network Policies Break Everything: You enabled network policies for security? Hope you enjoyed accessing Vault, because that's over now. Spent 8 hours debugging why Vault Agent Injector suddenly couldn't authenticate with Error: Get "https://kubernetes.default.svc:443/api/v1/tokenreviews": dial tcp: lookup kubernetes.default.svc on 10.96.0.10:53: no such host after our security team enabled Calico's default-deny policy. The webhook couldn't reach the Kubernetes API on port 6443 anymore. Calico and Cilium both have their own creative ways of fucking your setup.

Vault Downtime = CI/CD Downtime: When Vault's unavailable, your entire deployment pipeline stops. Better have a backup plan or very good Vault clustering. Raft storage helps with HA, but disaster recovery is still an Enterprise feature.

This shit works, but it's way more complex than HashiCorp admits.

Pick your poison. Here's what actually happens in production.

Kubernetes Service Account auth sounds simple until you realize it needs cluster-admin to set up properly. Your security team will love that.

Kubernetes auth method setup is straightforward - enable the auth method, create some RBAC, and pray the service account tokens don't rotate in the middle of your deployment. Here's what actually happens:

1. Your CI/CD pod gets a service account token (expires in 1 hour by default in new clusters)
2. Token gets sent to Vault for verification
3. Vault calls back to the Kubernetes API to validate it
4. If network policies are enabled, this fails mysteriously

Dynamic Secrets (The Good Part)

Database secrets are probably the only reason to use Vault. Supports PostgreSQL, MySQL, MongoDB, and dozens more. Instead of hardcoding postgres://user:password123@db:5432/app, your deployment requests temporary credentials that expire in 4 hours.

It actually creates real database users with specific permissions. When the TTL expires, that user disappears from the database. No cleanup scripts, no forgotten test credentials hanging around for months.

Database migrations get elevated privileges for schema changes, while normal app workloads get read-only or limited write permissions. Each deployment gets its own database user with a UUID suffix, so you can track exactly which deployment did what in your audit logs. Check out role-based credentials for fine-grained permissions.

Performance impact is real though - each secret request hits the database to create/revoke users. With 100 pods requesting secrets simultaneously during a rolling deployment, our Postgres 13 cluster started throwing FATAL: too many connections for role \"vault\" errors when CPU spiked to 95%. Postgres 14+ handles concurrent user creation much better, but you'll still want to bump max_connections from the default 100 to at least 200 unless you enjoy 3am pages about database connection exhaustion. We're still getting pages about connection limits weeks later even after the fix.

GitOps Reality Check

External Secrets Operator is your best bet for GitOps workflows. ESO polls Vault every few minutes and updates Kubernetes Secret objects. Works great until it silently stops working. Had ESO stop syncing secrets for like 3 weeks in production - pods kept starting with cached secrets until they expired, then maybe 47 services (could've been more) simultaneously started failing with Error: pq: password authentication failed for user \"vault_user_5f7a9b2c\" at something like 3:15 AM on a Tuesday. Felt like forever to fix. The ESO logs kept showing msg=\"successfully refreshed secret\" secret=api-keys the whole fucking time. Total lies.

Common ESO failure modes:

• Service account token expires (restart the operator)
• Network policies block Vault access (add proper ingress rules)
• Vault lease expires before refresh (tune your TTL values)
• SecretStore configuration gets out of sync with Vault policies

ArgoCD with vault-plugin fetches secrets during deployment, which is cleaner but breaks when Vault is unavailable. ArgoCD Vault Plugin replaces placeholders in your manifests with actual secrets from Vault. Alternative: Bank-Vaults or Helm Secrets for GitOps workflows.

The setup is painful and you'll spend hours debugging template syntax, but it keeps secrets out of Git completely.

When Vault Dies (And It Will)

Vault cluster outages take your entire CI/CD pipeline down. Plan accordingly:

• Agent caching helps but only for secrets you've already fetched
• Multiple Vault clusters require client-side failover logic
• Circuit breakers in your pipeline to fail fast instead of hanging
• Disaster recovery is Enterprise-only but critical for production

Network policies will break Vault connectivity in mysterious ways. Budget a full day for debugging when you enable them. The error messages are useless - everything just times out.

Database connection limits become a problem when every pod creates its own database user. Your Postgres max_connections setting suddenly matters a lot more. Consider connection pooling and credential caching strategies.

Monitoring That Actually Helps

Audit logs are great for compliance but terrible for debugging. Enable them for security, use metrics for operations:

• Secret request latency (alerts when > 5 seconds)
• Authentication failure rate (alerts when > 5%)
• Token expiration warnings (alerts 10 minutes before expiry)

Prometheus integration is straightforward and the metrics are actually useful. Set up alerts for the stuff that will page you at 3am:

• Vault unsealed status
• Secret request error rate
• Database connection exhaustion

SIEM integration works but you'll drown in logs. Focus on failed authentication attempts and secrets accessed outside business hours. Splunk, Datadog, and ELK stack integrations are all supported.

You're three errors deep and questioning your life choices by now. Here's what else will break.