Pod Security Admission Implementation Guide

Pod Security Policies got nuked in Kubernetes 1.25. Pod Security Admission is the replacement that actually works... most of the time.

PSA enforces security standards at the namespace level using simple labels. Unlike PSPs, which required you to have a fucking PhD in RBAC to understand what was happening, PSA runs as a built-in admission controller. This means PSA doesn't rely on those godawful webhooks that would randomly decide to take a shit during your Friday deployments.

PSA has been enabled by default since Kubernetes 1.23... or was it 1.22? Fuck it, I'm not looking it up right now - point is, if you upgraded past 1.20 something, it's probably there. Whether it's actually doing anything useful or just lurking in the background waiting to ruin your Friday deployment is anyone's guess.

Pod Security Standards Levels

PSA implements three security levels with increasing restrictions:

Privileged: No restrictions whatsoever. Your pods can do whatever the hell they want - mount the host, run as root, execute random binaries. Use this for system workloads and those ancient legacy apps that were built back when security wasn't even a thought.

Baseline: Blocks the really obvious security disasters like privileged containers and host networking. Most semi-modern apps can probably run under baseline without you wanting to throw your laptop out the window.

Restricted: Full paranoid mode. Pods must run as non-root with read-only filesystems and basically zero capabilities. Good luck getting anything from before 2021 to work with this - you'll be debugging security contexts until the heat death of the universe.

PSA Enforcement Modes

PSA supports three enforcement modes that can be applied independently to each namespace, which sounds flexible until you realize it's just three different ways for things to break:

• enforce: Actually rejects pods that violate the security standard. Failed deployments supposedly return "clear" error messages (spoiler: they don't, you'll still be scratching your head wondering wtf went wrong).
• audit: Lets pods run but creates logs that nobody reads. Great for compliance checkboxes.
• warn: Shows warnings to users who will promptly ignore them and deploy anyway.

You're supposed to start with audit and warn modes first to see what breaks, then enable enforcement once you've "fixed the obvious violations." Ha. What actually happens is you enable audit mode, get flooded with about 847 violations, realize your entire infrastructure was built on the assumption that everything runs as root, and then quietly go back to privileged mode.

Most teams enable audit mode at their target security level, spend about 3 days trying to make sense of the thousands of violations, give up, and just exempt everything. I may or may not have done this myself... multiple times.

Why Pod Security Policies Were Garbage

PSPs had so many fucking design issues that even the Kubernetes maintainers eventually said "fuck it, we're starting over":

• The RBAC relationships were so convoluted that figuring out which policy applied to which pod was like solving a Rubik's cube blindfolded
• Policy selection logic was about as transparent as a brick wall - pods would randomly get different policies and nobody knew why
• Every single policy needed its own special YAML file and a matching RBAC binding, because apparently complexity is a feature
• Error messages were written by someone who clearly hated users and wanted them to suffer

PSA fixes this clusterfuck by using explicit namespace labels - no more guessing games about which policy applies. But here's the catch: PSA trades PSP's fine-grained per-pod control for simpler namespace-level policies. Works fine for most people, but if you need different security settings for each pod in the same namespace, well... you're basically fucked.

PSA is easier to configure than PSPs, but that doesn't mean it's easy to implement. The namespace-level enforcement completely fucks over applications that need mixed security requirements in the same namespace, which is basically every real-world application.

How PSA Plays with Other Kubernetes Security

PSA works with other Kubernetes security stuff like security contexts, network policies, and RBAC. It validates pods early in the admission process, so you get immediate feedback when shit breaks instead of finding out later when your app mysteriously crashes.

PSA error messages are supposedly clearer than PSPs. When a pod gets rejected, the error is supposed to tell you what security control failed. "Supposed to" being the key phrase here - sometimes the errors are still cryptic as hell and about as helpful as a chocolate teapot.

I spent 6 fucking hours debugging why Prometheus node-exporter wouldn't start only to discover it needed to mount /proc from the host. The error message? "violates PodSecurity restricted:v1.29" - thanks for nothing, Kubernetes. Real helpful. I could have gotten more useful information from a Magic 8-Ball.

Here's some free advice: start with privileged mode everywhere and gradually tighten restrictions. Don't be like me and jump straight to restricted mode thinking you're some kind of security hero. I made that mistake once and spent the entire next Monday fixing 47 different apps that suddenly couldn't write to temp directories anymore. My coffee got cold about 15 times that day.

Is PSA Even Enabled? Let's Find Out The Hard Way

PSA supposedly comes enabled by default in Kubernetes 1.23+, but "enabled by default" in Kubernetes-land is about as reliable as a promise from a politician. Half the time it's there but not doing anything, and the other half it's secretly breaking your shit. Let me save you some time - here's how to check if you actually have it:

## This command will probably return nothing useful
kubectl get pods -n kube-system -l component=kube-apiserver -o yaml | grep PodSecurity

If that doesn't work (and it probably won't because nothing in Kubernetes is ever simple), try creating a privileged pod in a restricted namespace. If it fails, PSA is working. If it succeeds, PSA is either disabled, misconfigured, or just silently ignoring you because why the fuck not.

Cloud provider reality check (aka why managed Kubernetes isn't actually easier):

• EKS: Supposedly enabled in 1.23+, but this is AWS we're talking about - they somehow managed to make even the simple stuff complicated. At least it works most of the time.
• GKE: Works fine unless you're using Autopilot, in which case it becomes this weird mystery box where Google decides what's secure for you. Thanks Google, I definitely needed someone else making decisions about my cluster.
• AKS: Enabled by default and immediately starts fighting with Azure Policy like two drunk guys arguing about who's going to break your deployments first. Microsoft managed to create two competing security systems that hate each other.

The Namespace Label Dance

PSA uses namespace labels, which sounds simple until you actually try to implement it and discover all the ways it can fuck you over. Here's what the docs conveniently forget to mention:

apiVersion: v1
kind: Namespace
metadata:
  name: production-workloads
  labels:
    # This will break your legacy apps immediately
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: v1.29

    # This creates logs nobody reads
    pod-security.kubernetes.io/audit: baseline
    pod-security.io/audit-version: v1.29

    # This creates warnings developers ignore
    pod-security.kubernetes.io/warn: baseline
    pod-security.kubernetes.io/warn-version: v1.29

Gotcha #1: If your namespace name has special characters, PSA might completely shit the bed in mysterious ways. I learned this one the hard way with a namespace called "test-app_v2" that worked fine for months until I tried to enable PSA and it just... didn't. Spent 2 hours debugging before I tried renaming the namespace. Apparently underscores are the devil.

Gotcha #2: Version pinning is supposed to prevent surprises during cluster upgrades. In practice, you'll forget to update the versions for like 6 months, then during a Tuesday morning cluster upgrade to 1.28, you'll get mysterious errors about "unsupported PSA version" and spend 3 hours wondering why the fuck your monitoring pods won't start. Pro tip: that's when you discover you pinned everything to v1.24 and forgot to update it.

Version Pinning (Because Kubernetes Loves Breaking Changes)

Always pin your versions unless you enjoy surprises during cluster upgrades:

## Pin everything or suffer the consequences
pod-security.kubernetes.io/enforce-version: v1.29
pod-security.kubernetes.io/audit-version: v1.29
pod-security.kubernetes.io/warn-version: v1.29

What they don't tell you in the docs: if you don't pin versions, PSA will use whatever "latest" version means that week, which means your cluster upgrade might suddenly start enforcing new policies that nobody tested. This will 100% guaranteed break something important at 3 AM on a Friday when you're already three beers deep into your weekend. Trust me on this one.

Cluster-Wide Configuration (For Masochists)

You can set cluster-wide defaults, which sounds like a brilliant idea until you realize it applies to literally fucking everything and you can't take it back easily:

apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: PodSecurity
  configuration:
    apiVersion: pod-security.admission.config.k8s.io/v1beta1
    kind: PodSecurityConfiguration
    defaults:
      enforce: baseline  # Don't go straight to restricted
      enforce-version: v1.29
      audit: restricted
      audit-version: v1.29
      warn: restricted
      warn-version: v1.29
    exemptions:
      usernames: []
      runtimeClasses: []
      namespaces: [kube-system, kube-public, kube-node-lease]

WARNING: This configuration file goes in /etc/kubernetes/ on your control plane nodes. If you fuck up the YAML syntax even slightly, your entire cluster won't start and you'll be explaining to management why the production cluster is down because of a missing comma. Seriously, have backups and test this shit in dev first. I've seen people brick clusters with bad admission controller configs and it's not fun for anyone.

The Exemption List (Your Escape Hatch)

Some namespaces need to be exempt because they run system-level stuff that needs to break all the rules:

exemptions:
  namespaces:
    - kube-system          # Obviously
    - kube-public          # Ditto
    - kube-node-lease      # Don't touch this
    - istio-system         # Istio does what it wants
    - cert-manager         # Needs privileges for DNS challenges
    - monitoring           # Prometheus is basically root anyway
    - gitlab-runner        # CI/CD needs to do sketchy things

You'll discover more namespaces that need exemptions as things break. Keep a list.

The "Gradual Rollout" Lie (aka Why Documentation is Fiction)

The docs recommend this nice, neat phased approach that looks great in powerpoint slides. Here's what actually happens in the real world:

Phase 1 (Discovery):

kubectl label namespace production pod-security.kubernetes.io/audit=restricted

You'll get approximately 4,847 audit violations in the first hour. Nobody has time to read them all, and half of them will be duplicates anyway.

Phase 2 (User Education):

kubectl label namespace production pod-security.kubernetes.io/warn=restricted

Developers see warnings and ignore them. Nothing changes.

Phase 3 (Enforcement):

kubectl label namespace production pod-security.kubernetes.io/enforce=restricted

Everything breaks. You spend the next week debugging why your monitoring stack can't start.

Phase 4 (Reality):

kubectl label namespace production pod-security.kubernetes.io/enforce=privileged

You temporarily disable everything to get deployments working again.

Monitoring PSA (Or Trying To)

PSA generates events and logs that are supposed to help you track compliance:

## This returns way too much information
kubectl get events --field-selector reason=FailedCreate

## This might actually be useful
kubectl describe pod your-broken-pod

The error messages are cryptic at best. "violates PodSecurity restricted:v1.29" tells you absolutely fucking nothing about what specifically is wrong. You'll end up reading the pod spec line by line like you're looking for buried treasure, muttering "what the hell could it possibly be mad about now?" until you find that one random security context setting buried 30 lines deep.

What Actually Breaks

Here's what will definitely fail when you enable restricted mode:

1. Anything running as root (which is most legacy apps)
2. Pods without security contexts (basically everything from before 2020)
3. Init containers that need privileges (half your Helm charts)
4. Monitoring agents (they need host access)
5. CI/CD runners (they need to do privileged builds)
6. Legacy Java apps (they shit themselves without write access to temp directories)
7. Database containers (MySQL/PostgreSQL containers expect to create files as root)

Plan for at least 2-3 weeks of fixing broken deployments after enabling enforcement, and that's if you're lucky and don't have too much legacy garbage. I once spent 4 fucking hours debugging why Prometheus couldn't start, only to discover it needed write access to /tmp. The audit logs? They just told me "violates PodSecurity" - real helpful, thanks for nothing Kubernetes. I could have diagnosed the problem faster by reading tea leaves.

The Hard Truth About PSP to PSA Migration

Pod Security Policies got fucking nuked in Kubernetes 1.25, so if you're upgrading past 1.24, you don't have a choice in the matter. You're migrating whether you like it or not, and trust me, you won't like it.

The official migration guide makes it sound like this gentle, smooth transition. It's not. It's more like switching from Windows to Linux while blindfolded and on fire - everything you thought you knew is wrong and half your shit won't work.

Pre-Migration: Discovering the Damage

First, figure out what PSPs you actually have and whether they're doing anything:

## See what PSPs exist (probably none)
kubectl get psp

## Find out which pods actually use PSPs
kubectl get pods --all-namespaces -o custom-columns=\"NAMESPACE:.metadata.namespace,NAME:.metadata.name,PSP:.metadata.annotations.kubernetes\\.io/psp\"

Reality check: Most organizations discover they have PSPs configured but nothing is actually fucking using them. Turns out those years of PSP configuration were just elaborate security theater - all show, no substance.

If you do have active PSPs, mapping them to PSA levels is "fun":

• Your most restrictive PSP: Probably maps to baseline, not restricted (restricted is stricter than you think)
• Your "standard" PSP: Maps to privileged
• Your permissive PSP: Also maps to privileged

The Migration Process (And What Actually Happens)

Enable Audit Mode First

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: v1.29

What actually happens: You get about 10,000 audit violations in the first 30 minutes and realize every single goddamn pod in your cluster violates "restricted" policies. This is normal and expected, but it still feels like getting punched in the face.

Try to Analyze the Damage

## This returns way too much shit to be useful
kubectl get events --field-selector reason=PolicyViolation

## Check audit logs (if you can even find them)
journalctl -u kube-apiserver | grep \"pod-security\"

What you'll discover (spoiler alert, it's all bad news):

• Your entire monitoring stack runs as root because monitoring is basically spyware
• Every single init container needs privileges to do god knows what
• Half your Helm charts were written in 2018 by people who thought security was someone else's problem and just assume everything runs as root
• Your CI/CD pipeline is basically a privilege escalation playground that would make hackers weep with joy

Attempt Some Quick Fixes

You'll try to fix a few pods to be compliant:

## Before: Works
spec:
  containers:
  - name: app
    image: your-legacy-app:latest

## After: Doesn't work, app can't write to disk
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    fsGroup: 2000
  containers:
  - name: app
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        drop:
        - ALL

Result: Your app breaks because it can't write logs, create temp files, or do any of the normal things apps do. Check the security context docs for what actually works (spoiler: it's complicated).

Step 4: The Great Exemption

After a week of fighting with security contexts, you give up and exempt everything:

## The nuclear option
kubectl label namespace production pod-security.kubernetes.io/enforce=privileged
kubectl label namespace staging pod-security.kubernetes.io/enforce=privileged
kubectl label namespace development pod-security.kubernetes.io/enforce=privileged

Step 5: PSP Cleanup (Finally Something Easy)

## This probably fails because you don't have cluster admin
kubectl delete psp --all

## This might work
kubectl delete clusterrole psp:*
kubectl delete clusterrolebinding psp:*

Migration Challenges (The Real List)

Everything Runs as Root: Legacy applications, third-party containers, basically anything from before 2020 when security was just a vague suggestion. Fixing this requires rebuilding containers from scratch or just accepting that you'll have privileged namespaces everywhere.

Monitoring Agents Are Privileged by Design: Node exporters, log collectors, security scanners - they all need host access to spy on your system. Put them in exempt namespaces and move on with your life.

Init Containers: Every single fucking Helm chart has init containers that need to chown files or set permissions because apparently container images can't be built properly. These will all break under restricted policies. I spent an entire afternoon debugging why our PostgreSQL database wouldn't start after enabling baseline PSA - turns out the init container was trying to chown /var/lib/postgresql/data as user 0, which baseline blocks. The error? "violates PodSecurity baseline:v1.28: forbidden syscalls" - thanks for nothing, Postgres Helm chart. Four hours of my life I'll never get back tracing through initdb scripts.

Volume Mounts: Applications expect to write to /tmp, /var/log, and other paths. With readOnlyRootFilesystem: true, nothing works. Check the volume documentation for alternatives.

CI/CD is Fundamentally Incompatible: Docker-in-Docker, mounting Docker sockets, running privileged builds - CI/CD systems violate every security principle PSA enforces. Consider buildpacks or Tekton for more secure builds.

Troubleshooting: When Everything Breaks

Decoding PSA Error Messages

## The error message is useless
kubectl describe pod broken-app
## \"violates PodSecurity restricted:v1.29: forbidden syscalls\"

## What you actually need to check
kubectl get pod broken-app -o yaml | grep -A 50 securityContext

Common violations and their actual meanings:

• \"securityContext.runAsUser: 0\" = Your app runs as root, which is forbidden
• \"allowPrivilegeEscalation: true\" = Docker's default, blocked in restricted mode
• \"capabilities.add: [SYS_ADMIN]\" = Your app wants godmode privileges
• \"volumes: hostPath\" = Your app wants to mount host directories

Testing PSA Configuration

## Create a throwaway namespace for testing
kubectl create namespace psa-test
kubectl label namespace psa-test pod-security.kubernetes.io/enforce=restricted

## Try to deploy something simple
kubectl run test-pod --image=nginx --namespace=psa-test
## Watch it fail spectacularly

The Performance Lie

The docs claim PSA has "minimal performance impact." This is technically true - PSA itself is fast. But the troubleshooting, debugging, and constant namespace relabeling will consume weeks of engineering time.

Real performance impact:

• Engineering velocity: -50% for first month
• Time to deploy new apps: +300% (due to security context debugging)
• Incident response time: +200% (due to "is this a PSA issue?" questions)

Migration Timeline: Expectations vs Reality

What the fucking docs say: 6-8 weeks for complete migration (lol)
What actually happens in the real world:

• Week 1: Enable PSA, everything catches fire, nobody can deploy anything, all hands on deck to unfuck the situation
• Month 2-4: Fight with security contexts like you're wrestling a bear, maybe fix 20% of violations if you're lucky and have unlimited coffee
• Month 5-6: Give up on most apps, exempt 80% of namespaces to privileged mode, pretend this was always the plan
• Month 7+: New applications might get baseline enforcement if developers remember and if the stars align
• Month 12: Declare victory with 3 namespaces actually using baseline mode, update your resume

How to measure "success":

• What management wants to hear: "All applications running in restricted mode with zero security exceptions"
• What actually happens: "We stopped getting paged about PSA blocking deployments at 3 AM"
• What goes in the post-mortem: "PSA migration caused 4 production outages, 17 missed deployments, and 2 engineers to question their career choices before we exempted most workloads to privileged"