kube-bench - Actually Useful K8s Security Auditing

K8s security out of the box is a fucking joke. I'm talking `--anonymous-auth=true` enabled by default on kubelet until v1.10, etcd running without TLS encryption, API server bound to 0.0.0.0:8080 with zero authentication. The defaults scream "we care more about your developer experience than not getting breached."

kube-bench crawls through your cluster's guts and checks about 100 different ways your security is fucked. Learned this the hard way after spending 6 hours manually checking configs when a PCI audit flagged our payment processing cluster as "fundamentally insecure." The auditor literally said our k8s v1.8 cluster was "designed for attackers" because we had the default kubelet config from the getting started guide.

What Actually Gets Checked

Here's what kube-bench catches that will bite you in production:

API Server Shit That'll Get You Owned:

• `--anonymous-auth=true` still running (default until K8s v1.6)
• `--authorization-mode=AlwaysAllow` because someone was too lazy to set up RBAC
• Missing `--enable-admission-plugins=PodSecurityPolicy` (deprecated in v1.21 but still better than nothing)
• `--insecure-bind-address=0.0.0.0` - might as well post your cluster IP on 4chan

kubelet Fuckups That Keep Me Up at Night:

• Port `10255` wide open for anyone to scrape (disabled by default in v1.10, finally)
• `--allow-privileged=true` because someone needed to run a privileged pod "just once"
• `--anonymous-auth=true` on kubelet (why would you even do this?)
• Config file at `/var/lib/kubelet/config.yaml` with permissions 644 instead of 600 - because why not let everyone read your cluster secrets?

etcd - The Crown Jewel Everyone Forgets to Secure:

• `--client-cert-auth=false` because someone thought TLS was "too complicated"
• Data directory at `/var/lib/etcd` with 755 permissions - might as well make it a public FTP server
• Peer-to-peer traffic unencrypted because "it's internal network traffic, what could go wrong?"

Node-Level Ways to Get Pwned:

• Docker socket at `/var/run/docker.sock` mounted with 666 permissions because "containers need access"
• `vm.overcommit_memory=1` allowing memory bombs that can crash your nodes
• No AppArmor profiles because "they break stuff" and no seccomp because "too much work"

This motherfucker runs about 100 checks total. Takes maybe 30 seconds unless you're on a potato cluster. About 40% of checks fail on EKS because Amazon won't let you see master node configs (learned this after 3 hours of debugging "permission denied" errors).

The CIS Kubernetes Benchmark is basically a 300-page PDF of paranoid security requirements that auditors cum over.

There are like 5 different ways to run kube-bench, but most of them suck. Here's the one way that actually works without making you want to quit tech:

Kubernetes Job (The Only Way That Doesn't Suck)

Copy this YAML, apply it, pray to whatever deity you believe in:

apiVersion: batch/v1
kind: Job
metadata:
  name: kube-bench
spec:
  template:
    spec:
      hostPID: true
      nodeSelector:
        kubernetes.io/os: linux
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      containers:
      - name: kube-bench
        image: aquasec/kube-bench:latest
        command: ["kube-bench"]
        volumeMounts:
        - name: var-lib-etcd
          mountPath: /var/lib/etcd
          readOnly: true
        - name: var-lib-kubelet
          mountPath: /var/lib/kubelet
          readOnly: true
      volumes:
      - name: var-lib-etcd
        hostPath:
          path: "/var/lib/etcd"
      - name: var-lib-kubelet
        hostPath:
          path: "/var/lib/kubelet"

Run with: kubectl apply -f kube-bench-job.yaml && kubectl logs job/kube-bench

Shit That Will Ruin Your Day:

• `hostPID: true` makes security teams lose their minds, but without it you get "unable to retrieve processes" errors
• EKS users: 47 out of 100 checks will fail because Jeff Bezos doesn't trust you with master nodes
• Job hangs forever if nodes don't have `/var/lib/etcd` - learned this after watching it run for 2 hours on GKE
• Missing `restartPolicy: Never` means failed jobs restart infinitely and spam your logs

Direct Binary Installation (For Air-Gapped Hell)

If you're stuck in an air-gapped environment or need to run this directly on nodes, grab the latest release from GitHub:

## Download the binary (check releases page for latest version)
curl -L https://github.com/aquasecurity/kube-bench/releases/download/v0.12.0/kube-bench_0.12.0_linux_amd64.tar.gz -o kube-bench.tar.gz
tar xzf kube-bench.tar.gz
sudo mv kube-bench /usr/local/bin/
chmod +x /usr/local/bin/kube-bench

## Run it
sudo kube-bench --config-dir /etc/kube-bench/cfg

Error Messages That Made Me Rage-Quit:

• `permission denied: cannot access /proc/1234/cmdline` - run as root or kube-bench can't see kubelet args, wasted 2 hours on this obvious shit
• `Error: failed to load config from file "/etc/kubernetes/manifests"` - AWS/GCP hide this from you because they're control freaks, spent a Tuesday debugging this on EKS
• `no such file or directory: /var/lib/etcd` - managed clusters don't expose etcd, use `--benchmark eks` to skip these or enjoy 78 test failures
• `config not found: /etc/kube-bench/cfg` - path changed in v0.11.0, now you need --config-dir /opt/kube-bench/cfg, broke all my automation scripts
• `EACCES: permission denied, open '/var/lib/etcd/member/snap/db'` - the exact error that made me realize EKS etcd is in AWS la-la land

Cloud Platform Reality Check

EKS (Amazon): 47 out of 100 checks fail by design because Bezos thinks you can't handle master node access. Wasted 4 hours trying to figure out why etcd checks returned "no such file or directory" before someone told me AWS manages etcd in a black box. Use `--benchmark eks` or enjoy explaining to auditors why everything's broken.

AKS (Azure): 39 checks fail because Microsoft's Container-Optimized OS moves configs to /etc/kubernetes/azure.json instead of standard paths. Kubelet configs get 755 permissions by default, which kube-bench flags as insecure (it's right).

GKE (Google): Surprisingly doesn't suck. Only 12 checks fail on standard GKE, mostly around file permissions. Autopilot is a different beast - 78% failure rate because Google locked down everything and you can't change it.

Self-managed clusters: All checks work but you get to deal with actually fixing everything it finds. Which is... a lot.

Real CI/CD Integration

If you want this in your pipeline (and you should), output JSON and fail on critical findings:

kube-bench --json | jq -r '.Totals.total_fail' | xargs test 0 -eq

For AWS Security Hub integration, use `--asff` flag but expect to spend time mapping finding types to your compliance framework.

Companies use kube-bench because auditors demand it and because it catches the dumb shit that gets you fired when there's a breach. It's not sexy work, but it keeps the lights on and the lawyers happy.

Checkbox Checking for Fun and Profit

Regulated industries worship at the altar of CIS compliance. kube-bench spits out the magic reports auditors cream themselves over:

SOC 2 audits: Auditors see "automated security controls" and immediately get hard. Running kube-bench weekly shows you have "continuous monitoring" which sounds way more impressive than it is.

PCI DSS: Payment card industry auditors want proof your infrastructure doesn't suck. CIS compliance + kube-bench reports = checkbox checked, auditor happy.

HIPAA: Healthcare companies wave kube-bench reports at auditors to prove they're "implementing appropriate safeguards for patient data infrastructure."

FedRAMP/Government: Federal auditors demand NIST compliance, which has enough overlap with CIS that kube-bench reports satisfy most requirements.

Horror Story #1 - The Auditor Who Didn't Understand Cloud:
Auditor spent 3 hours grilling me about 47 "FAIL" results in our EKS kube-bench report. Dude was frantically taking notes about "critical control plane vulnerabilities" and "unauthorized etcd access." I'm trying to explain that AWS manages the master nodes and we literally can't access etcd even if we wanted to. He keeps pointing at test 1.1.12 (etcd data directory permissions) and asking why we haven't "remediated this critical finding." Finally brought in a Solutions Architect to explain that AWS doesn't give customers etcd access. Auditor's response: "So Amazon could be reading your encrypted secrets?" Face, meet palm.

Horror Story #2 - The Security Fix That Broke Everything:
Decided to "fix" all our kube-bench failures before the next audit. Set `--anonymous-auth=false` on kubelet to pass test 4.2.1. Deployed to prod on a Friday at 4:30pm (I know, I fucking know). Got paged at 2:47am because all our Grafana dashboards were blank - every single panel showing "No data." Prometheus couldn't scrape kubelet metrics on port 10255 because it was using anonymous auth - had been for 2 years. Spent until 6am rolling back the change, explaining to the very pissed on-call manager why "fixing security" broke monitoring, and writing the 3-page incident report. The kicker? That kubelet flag was the only way our monitoring worked, and nobody documented it because "anonymous auth is fine for internal metrics." Lost 4 hours of production monitoring because I trusted a security checklist over understanding what the fuck the config actually did.

The AWS Security Hub integration is useful if you're already sending compliance data there. Otherwise it's just another dashboard to ignore.

Real DevSecOps Integration

CI/CD pipelines: Run kube-bench on staging clusters before promoting to production. Parse the JSON output to fail builds on critical findings. Most teams only fail on high-severity issues because fixing everything is impossible.

Infrastructure validation: Use it to check Terraform/Helm deployments against CIS standards. Catches configuration mistakes before they hit production, but doesn't replace proper security review.

Scheduled monitoring: Weekly runs are common to catch configuration drift. Daily is overkill unless you're in a regulated industry with paranoid auditors who want reports every time someone breathes near the cluster.

Reality of Running at Scale

Multi-cluster hell: If you have dozens of clusters, you need automation to deploy and collect results. Expect different cloud providers to have different failure patterns.

Resource overhead: Uses almost no resources - maybe 50MB RAM and runs for 30 seconds. The network calls to check external endpoints sometimes time out on slow networks.

Parallel scaling: You can run multiple instances simultaneously but watch out for rate limiting if you're hitting the same etcd or API server endpoints.

Support and Maintenance

Open source support: Active community on GitHub for issues and feature requests. Maintainers are responsive but don't expect 24/7 support for free.

Commercial support: Aqua Security offers paid support if you need SLAs and guaranteed response times. Most companies don't need this unless they're risk-averse enterprises.

Updates: Tool gets updated regularly for new Kubernetes and CIS benchmark versions. Usually lags a few months behind but catches up eventually.

Enterprise features: Aqua's commercial platform adds remediation guidance and integration with other security tools. Useful if you're already paying for their other products.

The AWS Security Hub integration is useful if you're already sending compliance data there.