Pod Security Standards: AI-Optimized Implementation Guide

Critical Context and Migration Reality

Migration Timeline: Plan 2-3 months for clusters with 100+ applications

• Week 1: Enable audit mode, discover violations (expect 300+ issues across 80+ deployments)
• Weeks 2-8: Fix applications while developers complain
• Weeks 9-12: Fix production-only edge cases
• Month 4+: Enable enforce mode, discover missed edge cases
• Ongoing: Handle failures during routine updates and node reboots

Key Pain Points:

• Pod Security Policies (PSP) deprecated in K8s 1.21, removed in 1.25
• Error message "violates PodSecurity restricted" provides no actionable detail
• Legacy applications assume root access and filesystem write permissions
• Managed services (EKS, GKE, AKS) have undocumented platform-specific gotchas

Security Profile Configuration

Three Security Levels

Profile	Use Case	Key Restrictions	Common Failures
Privileged	System namespaces (kube-system)	None - allows everything	None (intended for system pods)
Baseline	Most practical applications	Blocks privileged containers, host namespaces, dangerous capabilities	Minimal - most Helm charts from 2019+ work with minor patches
Restricted	Production applications	Non-root execution required, all capabilities dropped except NET_BIND_SERVICE, mandatory seccomp/AppArmor	High failure rate - breaks monitoring agents, DNS tools, init containers, legacy Java apps

Critical Implementation Requirements

Namespace Configuration:

apiVersion: v1
kind: Namespace
metadata:
  name: production-apps
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted  
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/enforce-version: v1.29

Enforcement Modes:

• enforce: Kills non-compliant pods (use carefully in production)
• audit: Logs violations to audit logs
• warn: Shows warnings to kubectl users

Version Pinning Gotcha: Pinning to older versions (e.g., v1.24) during K8s upgrades prevents new security features from working

Common Application Failures with Restricted Mode

High-Probability Failures:

• Prometheus node-exporter: Requires hostNetwork: true
• DNS-based service discovery: Needs NET_ADMIN capability, fails silently
• Legacy Java applications: Write to /tmp as root
• Init containers: Configure file permissions, break consistently
• Istio sidecars: Pre-2020 versions break extensively
• Database backup scripts: Assume root filesystem access
• Fluent Bit logging agents: Need to read /var/log/containers as root
• NGINX Ingress Controller: Sidecars can't bind to port 80
• MySQL init containers: chown data directories

Root Cause Categories:

1. Root execution assumption: Applications written before 2020
2. Privileged container requirements: System-level access needs
3. Filesystem permission dependencies: Write access to system directories
4. Network capability requirements: Binding to privileged ports, network management

Managed Service Platform Gotchas

AWS EKS:

• ALB Controller requires undocumented special permissions
• AWS-specific security configurations not in standard documentation

Google GKE:

• Autopilot enforces its own Pod Security standards that conflict with custom settings
• Platform networking quirks affect seccomp profile restrictions

Azure AKS:

• Azure CNI requires specific configurations for security features
• Network policies interact unpredictably with Pod Security Standards

Migration Strategy and Troubleshooting

Recommended Approach:

1. Start with audit mode only - never skip this step
2. Enable warn mode for developer feedback
3. Parse audit logs for violations (grep for "violates" in JSON logs)
4. Namespace-by-namespace rollout: Dev → Staging → Production
5. Test with CI/CD pipeline before production deployment

Debugging Techniques:

• Use kubectl describe pod for slightly better error information
• Enable warn mode for real-time feedback during deployment
• Audit logs are verbose - requires extensive parsing for actionable data
• Common debugging requires guessing which security control failed

Emergency Recovery:

Nuclear option: kubectl delete namespace && kubectl create namespace clears Pod Security labels

• Always backup configurations first
• Required when namespaces enter inconsistent states during testing

Resource Requirements and Costs

Technical Resources:

• Performance impact: ~50ms pod startup delay for admission controller checks
• Developer time: Weeks debugging securityContext configurations
• Operational overhead: Extensive log parsing and troubleshooting

Expertise Requirements:

• Kubernetes security knowledge: Understanding of capabilities, seccomp, AppArmor
• Application architecture: Knowledge of legacy application requirements
• Platform-specific knowledge: Cloud provider security implementations

Integration with Security Ecosystem

Complementary Tools:

• Network Policies: Control traffic (separate from Pod Security Standards)
• RBAC: API access control (independent layer)
• Service meshes: Additional security and complexity (Istio, Linkerd)

Advanced Policy Engines:

• OPA Gatekeeper: Custom Rego policies, steep learning curve, debugging difficulties
• Kyverno: Easier than Gatekeeper, YAML-based policies
• Kubewarden: WebAssembly-based, newer ecosystem
• Polaris: Scanning and validation webhooks, simpler alternative

Tool-Specific Intelligence:

• Gatekeeper: 6-month trial period common before abandonment due to complexity
• Polaris: Finds 300+ violations in typical first scan, many false positives
• Falco: Runtime security monitoring, integrates with Pod Security Standards

Production Readiness Checklist

Pre-Migration Requirements:

• Audit existing Pod Security Policies and application dependencies
• Identify applications requiring privileged access with business justification
• Test in non-production environments with identical workloads
• Establish monitoring for Pod Security violations in audit logs
• Document remediation procedures for common failure scenarios

Post-Migration Monitoring:

• Monitor pod startup failure rates
• Track application functionality regression
• Audit security compliance across all namespaces
• Verify backup and disaster recovery procedures still function
• Test cluster upgrade procedures with Pod Security Standards enabled

Critical Warnings

What Official Documentation Doesn't Tell You:

• Breaking point: Most organizations discover 40+ applications running as root
• Silent failures: DNS and networking issues may not surface immediately
• Version compatibility: K8s upgrade procedures change with Pod Security Standards
• Platform variations: Each cloud provider implements subtle differences
• Debugging difficulty: Error messages provide minimal actionable information

Failure Scenarios and Consequences:

• Production downtime: Enabling enforce mode without thorough testing
• Monitoring blindness: Security agents failing during incident response
• Backup failures: Database and application backup scripts breaking
• Service discovery outages: DNS-based discovery failing silently
• Application regression: Legacy functionality depending on elevated privileges

This guide provides operational intelligence for successful Pod Security Standards implementation while avoiding common pitfalls that cause production incidents and extended migration timelines.