You've got the basics locked down - RBAC, network policies, pod security standards. Now it's time for the advanced shit that separates production-ready clusters from the ones that get owned.
Admission Controllers: The Last Line of Defense
Admission controllers are your final chance to block dangerous deployments before they hit your cluster. Think of them as automated security reviews that never get tired, never approve shit just to make a deadline, and never forget to check for privilege escalation.
OPA Gatekeeper Implementation
Install Gatekeeper and prepare for some YAML hell:
## Install Gatekeeper
kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/release-3.14/deploy/gatekeeper.yaml
## Wait for it to be ready (this takes a few minutes)
kubectl wait --for=condition=Ready pod -l control-plane=controller-manager -n gatekeeper-system --timeout=300s
Create constraint templates for common security violations:
## privileged-containers-template.yaml
apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
name: k8srequiredsecuritycontext
spec:
crd:
spec:
names:
kind: K8sRequiredSecurityContext
validation:
properties:
runAsNonRoot:
type: boolean
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package k8srequiredsecuritycontext
violation[{"msg": msg}] {
container := input.review.object.spec.template.spec.containers[_]
not container.securityContext.runAsNonRoot
msg := "Container must run as non-root user"
}
violation[{"msg": msg}] {
container := input.review.object.spec.template.spec.containers[_]
container.securityContext.privileged
msg := "Privileged containers are not allowed"
}
Apply the constraint:
## privileged-containers-constraint.yaml
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredSecurityContext
metadata:
name: must-run-as-nonroot
spec:
match:
kinds:
- apiGroups: ["apps"]
kinds: ["Deployment"]
excludedNamespaces: ["kube-system", "gatekeeper-system"]
parameters:
runAsNonRoot: true
Critical gotcha: Gatekeeper has no dry-run mode. Test constraints in a staging cluster first, or you'll block legitimate deployments.
Custom Admission Webhooks
For complex policies, write custom admission webhooks. Here's a simple webhook that blocks images from untrusted registries:
// admission-webhook.go
package main
import (
"context"
"encoding/json"
"fmt"
"net/http"
"strings"
admissionv1 "k8s.io/api/admission/v1"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
var trustedRegistries = []string{
"registry.company.com",
"gcr.io/company-project",
"docker.io/library", // Official Docker images
}
func admissionHandler(w http.ResponseWriter, r *http.Request) {
var body []byte
if r.Body != nil {
body, _ = io.ReadAll(r.Body)
}
var admissionReview admissionv1.AdmissionReview
json.Unmarshal(body, &admissionReview)
req := admissionReview.Request
var pod corev1.Pod
json.Unmarshal(req.Object.Raw, &pod)
allowed := true
message := ""
for _, container := range pod.Spec.Containers {
if !isImageTrusted(container.Image) {
allowed = false
message = fmt.Sprintf("Image %s from untrusted registry", container.Image)
break
}
}
response := &admissionv1.AdmissionResponse{
UID: req.UID,
Allowed: allowed,
Result: &metav1.Status{Message: message},
}
admissionReview.Response = response
respBytes, _ := json.Marshal(admissionReview)
w.Header().Set("Content-Type", "application/json")
w.Write(respBytes)
}
func isImageTrusted(image string) bool {
for _, registry := range trustedRegistries {
if strings.HasPrefix(image, registry) {
return true
}
}
return false
}
Deployment reality: Custom webhooks are powerful but require TLS certificates, proper error handling, and monitoring. Budget 2-3 weeks for production-ready webhook implementation.
Runtime Monitoring: Catching Active Attacks
Static policies catch deployment mistakes. Runtime monitoring catches actual attacks in progress.
Falco Setup and Configuration
Install Falco with proper rule customization:
## Install Falco using Helm
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
## Create custom values file
cat << 'EOF' > falco-values.yaml
falco:
rules_file:
- /etc/falco/falco_rules.yaml
- /etc/falco/falco_rules.local.yaml
- /etc/falco/k8s_audit_rules.yaml
- /etc/falco/rules.d
# Reduce noise - default rules are too aggressive
load_plugins: [k8saudit, json]
# Output to multiple destinations
json_output: true
json_include_output_property: true
# File output for log aggregation
file_output:
enabled: true
keep_alive: false
filename: /var/log/falco.log
# gRPC output for real-time alerts
grpc:
enabled: true
bind_address: "0.0.0.0:5060"
threadiness: 8
## Custom rules to reduce false positives
customRules:
rules.yaml: |
# Override default rules with production-tested versions
- rule: Write below binary dir
desc: Detect file writes below common binary directories
condition: >
bin_dir and evt.dir = < and open_write
and not package_mgmt_procs
and not exe_running_docker_save
and not python_running_get_pip
and not python_running_ms_oms
and not user_known_write_below_binary_dir_activities
and not nodejs_running_npm
# Add your application exceptions here
and not proc.name in (your-app-binary)
output: >
File below a known binary directory opened for writing
(user=%user.name command=%proc.cmdline file=%fd.name)
priority: ERROR
tags: [filesystem, mitre_persistence]
EOF
helm install falco falcosecurity/falco -f falco-values.yaml -n falco --create-namespace
Critical tuning step: Falco will spam the hell out of you for the first week. Create custom rules to filter out your application's normal behavior:
## falco-custom-rules.yaml
- rule: Allow expected file writes
desc: Allow your application to write expected files
condition: >
(proc.name = your-app-name and fd.name startswith /app/data)
or (proc.name = nginx and fd.name startswith /var/log/nginx)
or (proc.name = postgres and fd.name startswith /var/lib/postgresql)
priority: INFO
- list: known_drop_and_execute_containers
items: [your-ci-runner, your-build-container]
## Override default rule to exclude your containers
- rule: Drop and execute new binary in container
condition: >
spawned_process and container and
(proc.is_exe_upper_layer = true or proc.is_exe_from_memfd = true) and
not container.image.repository in (known_drop_and_execute_containers)
append: false
Alert Management and Response
Connect Falco to your incident response system. Here's a webhook forwarder for Slack/PagerDuty:
## falco-sidekick-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: falco-sidekick-config
namespace: falco
data:
config.yaml: |
slack:
webhookurl: "https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK"
channel: "#security-alerts"
username: "falco"
minimumpriority: "error"
pagerduty:
routingkey: "your-pagerduty-integration-key"
minimumpriority: "critical"
# Custom webhook for your SIEM
webhook:
address: "https://your-siem.company.com/api/events"
minimumpriority: "warning"
Response playbook: When Falco fires alerts, don't just dismiss them. Follow this process:
- Immediate: Check if it's a known false positive
- 5 minutes: Verify the pod/container is legitimate
- 15 minutes: Check network connections from the suspicious pod
- 30 minutes: Review audit logs for privilege escalation attempts
Runtime Security Beyond Falco
eBPF-based monitoring for deeper system visibility:
## Install Tetragon for network and process monitoring
helm repo add cilium https://helm.cilium.io/
helm install tetragon cilium/tetragon -n tetragon --create-namespace \
--set tetragon.enableProcessCred=true \
--set tetragon.enableProcessNs=true
File integrity monitoring using AIDE:
## aide-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: aide-fim
namespace: security
spec:
selector:
matchLabels:
name: aide-fim
template:
spec:
hostNetwork: true
hostPID: true
containers:
- name: aide
image: your-registry.com/aide:latest
securityContext:
privileged: true
volumeMounts:
- name: host-root
mountPath: /host
readOnly: true
command: ["/usr/bin/aide"]
args: ["--check", "--config=/etc/aide.conf"]
volumes:
- name: host-root
hostPath:
path: /
Putting It All Together
Your final security stack should include:
- Prevention: RBAC, network policies, pod security standards, admission controllers
- Detection: Runtime monitoring, file integrity, network analysis
- Response: Automated alerts, incident playbooks, forensic capabilities
Operational overhead: Plan for 10-15 hours/week managing security tooling. More during initial tuning, less once it's stable.
Performance impact: Properly configured security adds 5-10% CPU overhead and 50-100ms request latency. The alternative is getting owned, so it's worth it.
Testing regimen: Run chaos engineering exercises monthly. Simulate compromised pods, privilege escalation attempts, and data exfiltration. If your security tools don't catch it in testing, they won't catch it during a real attack.
The next step is compliance frameworks and audit preparation, but nail the runtime monitoring first. You can't secure what you can't see.