Pod Security Standards - Three Security Levels Instead of Policy Hell

The PSP to Pod Security Standards migration was supposed to be simple. It wasn't. Half our legacy apps broke because they were running as root and nobody documented why. But compared to the YAML hell of Pod Security Policies, which got deprecated in K8s 1.21 and finally killed in 1.25, Pod Security Standards is actually a relief.

Instead of writing custom PSP YAML files that nobody understood (we all copy-pasted from Stack Overflow), you get three options. Done. Pod Security Admission handles everything through namespace labels. Actually stable since 1.25 and built right into K8s, which is nice for once. Still need to configure it properly, but at least it exists without needing another third-party operator.

The Three Security Profiles

Privileged is basically "fuck it, do whatever you want" mode. Everything's allowed - privileged containers, host network access, hostPath volumes. Your kube-system namespace probably needs this because system pods are special snowflakes that need to touch everything.

Baseline blocks the obvious stupid shit while still being practical. No privileged containers, no host namespace sharing, no dangerous Linux capabilities. Most Helm charts from 2019 work with Baseline after you patch a few securityContext settings.

Restricted is where things get real. Non-root execution required, all capabilities dropped except NET_BIND_SERVICE, mandatory seccomp and AppArmor profiles. This breaks more things than you expect - DNS-based service discovery, init containers that write configs, monitoring agents that need special access. We had three apps that took weeks to fix because they assumed they could write to /tmp as root.

Implementation Through Pod Security Admission

Pod Security Admission is built into K8s now (finally stable in 1.25) but of course you still need to configure it. Three modes: enforce (kills pods that don't comply), audit (logs violations), and warn (shows warnings to kubectl users).

Start with audit and warn modes before enabling enforce - seriously, don't skip this step or you'll take down production. EKS, GKE, and AKS all support Pod Security Standards now, but the configuration varies enough to make you swear at your cloud provider.

Namespace-level labels control everything. Your kube-system namespace stays Privileged, production apps get Restricted, everything else gets Baseline. The error messages are useless - "violates PodSecurity restricted" is the most unhelpful error message in K8s. It doesn't tell you WHICH control failed, so you get to play guessing games.

Migration from Pod Security Policies

The migration from PSPs is like ripping off a Band-Aid that's been stuck for three years. Google's migration guide makes it sound easy - surprise, it's fucking not.

First, you audit your existing PSPs to see what they actually do. Most organizations discover they have 10 different PSPs that do basically the same thing, plus one snowflake policy for that legacy Java app that needs root access for "historical reasons" (nobody remembers what those reasons were).

The real timeline: Plan 2-3 months for a cluster with 100+ apps if you want to sleep at night. The policy change takes 5 minutes, fixing everything that breaks takes forever. Our first production cluster migration? Disaster. Found like 40-something apps running as root (maybe 47? stopped counting), a bunch that needed privileged containers because "legacy reasons" that nobody could explain, and one critical database backup script that just assumed it owned the entire filesystem. Version pinning seems smart until you're stuck on an old standard during cluster upgrades and can't figure out why your new deployments aren't starting. We learned this the hard way during a K8s 1.27 upgrade when half our apps wouldn't restart.

Pod Security Standards work through namespace labels that tell the admission controller what to do. Simple concept, fucking annoying in practice when you've got 50 namespaces that all need different settings.

Namespace Configuration

Three namespace labels per profile. Copy this and adjust to taste:

apiVersion: v1
kind: Namespace
metadata:
  name: production-apps
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted  
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/enforce-version: v1.29

Enforce kills pods that don't comply - use this carefully or you'll take down production wondering why nothing fucking starts. Audit logs violations to your audit logs (hope you're actually reading them). Warn shows kubectl users what's wrong - helpful for developers who don't read documentation anyway.

Version Pinning and Compatibility

Version pinning with the -version suffix sounds like a good idea until you realize you're pinned to like v1.24 standards during a K8s 1.29 upgrade and wondering why new security features aren't working. I spent two days debugging why our new seccomp profiles weren't applying before realizing we were still pinned to old standards. Felt pretty dumb about that one.

Without version pinning, you get the latest standards for your K8s version. This usually works fine unless you're upgrading across multiple K8s versions and something breaks. The compatibility matrix exists but reading it requires more coffee than most people have available.

Cluster-Level Configuration

Cluster-wide defaults through AdmissionConfiguration if you want to set policy once instead of labeling 47 namespaces individually:

apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: PodSecurity
  configuration:
    apiVersion: pod-security.admission.config.k8s.io/v1
    kind: PodSecurityConfiguration
    defaults:
      enforce: baseline
      audit: restricted
      warn: restricted
    exemptions:
      usernames: ["system:serviceaccounts:kube-system"]
      namespaces: ["kube-system", "kube-public"]

This sets baseline enforcement with restricted auditing everywhere except exempted namespaces. Your kube-system needs exemptions or system pods won't start and you'll wonder why your cluster is broken. Trust me on this one - I spent half a day debugging why our control plane wouldn't start after enabling Pod Security Standards. Not my finest moment.

Migration Strategies

Start with audit mode - seriously, don't fucking skip this. Enable audit and warn first, then spend a week reading logs to see what breaks. Audit logs are verbose garbage filled with every syscall and capability check, but good luck finding actual violations without grepping for hours. You'll be parsing JSON logs looking for "violates" while wondering why Kubernetes makes everything harder than it needs to be.

Namespace-by-namespace works if your teams don't hate each other. Dev namespaces get restricted first (devs love complaining about security changes), staging gets it next (where you discover what really breaks), production gets it last (after you fix everything twice).

Speaking of managed services - EKS users get the bonus of learning AWS-specific gotchas, like how ALB controller needs special permissions that nobody documents properly. Don't get me started on GKE Autopilot, which quietly enforces its own Pod Security standards that conflict with yours in subtle ways. And Azure? Azure CNI has its own networking quirks that break when seccomp profiles get too restrictive.

Automated checking sounds great until your CI/CD pipeline starts failing because that Helm chart you copied from the internet assumes root access. Polaris and Falco help but they'll find way too many violations you didn't know existed. Our first scan? Jesus, I think it was like 300+ issues spread across 80-something deployments. Most turned out to be false positives but sorting through that mess took over a week. Still have PTSD from that JSON output.

Integration with Security Tools

Pod Security Standards don't replace your existing security tools - they just add another layer to debug when things break. Network Policies still control traffic, RBAC still controls API access. Service meshes like Istio add even more security (and complexity).

Third-party engines like OPA Gatekeeper, Kyverno, and Kubewarden let you write custom policies on top of Pod Security Standards. Great for organizations that need more than three security profiles or love writing Rego policies that nobody can debug when they break. We tried Gatekeeper for six months before giving up - the learning curve is steeper than K8s itself and debugging policies feels like archeology.

When Things Break (They Will)

"violates PodSecurity restricted" is the most useless error message in K8s. It doesn't tell you which control failed. Use kubectl describe on the pod to see slightly more helpful information, or enable warn mode to get feedback during deployment. Still not great, but better than guessing.

Things that usually break with Restricted:

• Prometheus node-exporter needs hostNetwork: true - took me like 4 hours to figure this out, docs don't really mention it
• DNS-based service discovery containers often need NET_ADMIN capability - ours failed silently for days before we caught it
• Legacy Java apps that write to /tmp as root - our Spring Boot apps all did this, took weeks to patch them all
• Init containers that configure file permissions - most of ours broke, including cert-manager
• Older sidecars from before 2020 - istio-proxy was a nightmare, ended up upgrading the entire mesh
• That one Postgres backup script that assumed root access - broke our nightly backups for over a week
• Fluent Bit logging agents need to read /var/log/containers as root - discovered this during a production incident, of course
• NGINX Ingress Controller sidecars break if they can't bind to port 80 - obvious in hindsight, painful in reality
• MySQL init containers that chown data directories - most Helm charts did this, broke pretty consistently

The nuclear option: kubectl delete namespace && kubectl create namespace clears Pod Security labels if you're desperate. Just remember to back up your configs first. I've had to do this twice when namespaces got into weird states during testing.