cert-manager - Stops You From Getting Paged at 3AM Because Certs Expired Again

I discovered cert-manager after our SSL certs expired on Black Friday weekend and killed our entire e-commerce site for 4 hours. Nothing like watching revenue tank while you fumble with Let's Encrypt CLI tools at 3am, trying to explain to the CEO why "certificate expired" means customers can't buy anything.

cert-manager saves your ass when certificates expire and you're the one getting paged. Jetstack created it in 2016 after getting tired of the same certificate management nightmare we all face. The CNCF graduated it on November 12, 2024 because literally everyone was using it anyway - might as well make it official.

The Three Things That Actually Matter

Certificate Resources: You define what domains need certs using Kubernetes custom resources. cert-manager watches these and handles the renewal dance automatically through the ACME protocol. Set it once, forget it exists until something breaks (which it rarely does).

Issuer/ClusterIssuer Resources: Point to your certificate authority. Let's Encrypt for public stuff, HashiCorp Vault PKI for internal certificates, or whatever enterprise CA your security team is obsessing over this quarter. ClusterIssuer works cluster-wide; Issuer is namespace-scoped.

CertificateRequest Resources: The actual certificate signing requests that follow the X.509 standard. You usually don't touch these - cert-manager creates them automatically when certificates need renewal. Only mess with these if you know what you're doing or debugging failed certificate issuance.

The Numbers Don't Lie (Because Everyone's Been Burned)

The 500+ million monthly downloads sound like marketing bullshit but they're real - it's because we've all been there. Your site goes down because some certificate you forgot about expired. Kubernetes adoption surveys show 86% of production clusters run cert-manager because manual cert renewal is like playing Russian roulette with production.

Real talk: I've personally seen Let's Encrypt rate limits fuck over teams who waited until the last minute to renew 20+ domain certs. That 50 certificates per domain per week limit hits hard when you're scrambling.

What Works (And What Doesn't)

HTTP-01 challenges work great until your ingress controller decides to shit the bed. Let's Encrypt tries to validate ownership by hitting yourdomain.com/.well-known/acme-challenge/some-token and if that returns 404 or times out, you're fucked. I've spent hours debugging ingress-nginx configuration just to get ACME working again.

DNS-01 challenges are your friend for internal services and wildcard certs, but DNS provider APIs are consistently terrible. cert-manager creates TXT records like _acme-challenge.example.com to prove domain ownership. Works with Route53, Cloudflare, Google Cloud DNS, and 50+ other providers. DNS propagation can take forever though - GoDaddy is especially slow as hell.

The latest version 1.18.2 from July 2, 2025 fixes private key rotation edge cases that caused certificates to randomly fail validation. Nothing revolutionary, just fewer "why the fuck did this break" moments. Always check the upgrade guide - cert-manager migrations have burned me before when webhook configurations changed between versions.

Installation Reality Check

Skip the kubectl apply bullshit with raw YAML - that path leads to webhook validation failures and hours of debugging. The Helm chart actually works:

helm repo add jetstack https://charts.jetstack.io
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set installCRDs=true

That --set installCRDs=true flag is critical - without it you have to manually install custom resource definitions and trust me, you'll fuck up the order. I spent 6 hours debugging webhook failures because I forgot this flag. The installation docs bury this detail for some reason.

You get three pods that actually do stuff: the main controller handles certificates, a validating webhook validates your YAML, and a CA injector patches webhook configurations with CA bundles. Resource requirements are reasonable - about 50MB RAM and 10m CPU each in normal operation. Scales well unless you're managing thousands of certificates across multiple clusters.

Multi-Issuer Setup (The Right Way)

Production environments need multiple certificate sources. Here's what actually works:

Let's Encrypt for public services - free, automated, works everywhere:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ops@yourcompany.com  # They send expiration warnings here
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          class: nginx  # or traefik, whatever you use

Internal CA for internal services - because security teams love internal PKI:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: ca-issuer
spec:
  ca:
    secretName: ca-key-pair

Staging environment - use Let's Encrypt staging for testing to avoid rate limits:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory

DNS-01 Challenges (When HTTP-01 Won't Work)

Internal services and wildcard certificates need DNS-01. Route53 integration works best:

spec:
  acme:
    solvers:
    - dns01:
        route53:
          region: us-east-1
          accessKeyID: AKIAIOSFODNN7EXAMPLE  # Use IRSA instead
          secretAccessKeySecretRef:
            name: route53-credentials
            key: secret-access-key

Pro tip: Use IAM Roles for Service Accounts (IRSA) instead of hardcoded AWS access keys. Your security team will thank you, and AWS security best practices recommend avoiding long-term credentials.

Service Mesh Integration (If You Must)

istio-csr replaces Istio's built-in certificate management with cert-manager. Useful for consistent certificate policies across service mesh and ingress:

helm install -n istio-system cert-manager-istio-csr jetstack/cert-manager-istio-csr

It handles mTLS certificates for service-to-service communication. Works fine but adds complexity - only use if you need unified certificate management.

What Burns Down Production (Learn From My Pain)

Resource limits will crash cert-manager under load - the default resource limits are pathetically small. Our cert-manager died during a mass certificate renewal, took us 3 hours to figure out why:

resources:
  limits:
    cpu: 200m      # Default 100m is not enough 
    memory: 256Mi   # Default 128Mi will OOM under pressure
  requests:
    cpu: 50m       # Be realistic about CPU needs
    memory: 64Mi

Webhook timeouts kill everything - the admission webhook has a 10-second timeout that's too short when DNS is being slow. I've seen DNS-01 challenges fail because DNS propagation took 15 seconds:

webhook:
  timeoutSeconds: 30  # Give DNS time to not suck

Let's Encrypt rate limits are brutal - 50 certificates per registered domain per week. We hit this limit during a Kubernetes migration and couldn't get new certs for 3 days. Plan your certificate requests or you're fucked.

DNS propagation delays - DNS-01 challenges fail randomly when DNS changes take forever to propagate. cert-manager retries, but sometimes you have to manually kick your DNS provider. Cloudflare DNS is usually fast, GoDaddy DNS can take 30+ minutes for no good reason.

Monitoring (Because Things Break)

Prometheus metrics tell you when renewals fail. This alert saved my ass when our API gateway cert was about to expire during Christmas break:

- alert: CertManagerCertificateExpirySoon
  expr: cert_manager_certificate_expiration_timestamp_seconds - time() < 86400 * 7
  for: 1h
  annotations:
    summary: "Certificate expires in less than 7 days"

Monitor cert_manager_certificate_renewal_timestamp_seconds and cert_manager_acme_client_request_count to catch issues early. Set up alerts for failed renewals too - trust me, you don't want to discover cert-manager is broken when customers start complaining.

Security Hardening

RBAC permissions are overly broad by default. Version 1.18 lets you disable HTTP-01 challenges if you only use DNS-01:

global:
  disableHTTP01Solver: true

Private key storage - keys go in Kubernetes Secrets by default. For paranoid environments, use the CSI driver for ephemeral keys that never touch etcd.

Certificate approval - approver-policy adds human approval workflows for regulated environments. Slows things down but compliance teams love it.