Hardening GKE Enterprise - Security That Actually Works

I've debugged enough cluster breaches to know they all start the same way: "Our security is fine, we're using GKE." Standard GKE defaults assume you'll configure everything properly. Spoiler alert: you won't. Nobody fucking does.

The Reality Check Nobody Talks About

Here's what actually happens when you run production workloads on basic GKE:

Service account keys everywhere. Every team has their own special snowflake way of mounting JSON keys. Half are committed to Git, the other half are in Slack channels because "it's just temporary." These keys never rotate, never expire, and when one gets compromised, you're dealing with credential stuffing attacks for months. Every breach I've worked on starts with some leaked service account key that had cluster-admin or close to it.

Container images from random Docker Hub repos. Someone needed Redis, grabbed the first image that looked official, and now you're running god-knows-what in production. No scanning, no signing, no idea what's actually in that container until it starts mining bitcoin at 2 AM on a Sunday. I once debugged a cluster breach where the attacker got in through a Redis container named "redis-official-best" from Docker Hub. Turned out it was crypto mining malware that had been downloaded 50,000 times before Docker finally pulled it down.

Network policies that don't exist. Everything can talk to everything because "network policies are too complex." Your database is one misconfigured pod away from being public to the internet. Most teams I work with have been hit by something - credential leaks, crypto miners in containers, or attackers moving between pods because nothing blocks lateral movement.

RBAC that makes no sense. Half your pods run as cluster-admin because "it was easier than figuring out the actual permissions." Security through obscurity isn't security—it's just temporary ignorance. Every cluster I audit has at least one service account with cluster-admin that definitely shouldn't. Found one cluster where the monitoring service had cluster-admin because someone couldn't figure out why it couldn't read metrics. Turns out it just needed get on nodes/metrics.

Every cluster breach I've worked on follows the same pattern:

• Compromised service account with too many permissions
• Lateral movement through overprivileged pods
• Data exfiltration or crypto mining (sometimes both)
• Months of cleanup and explaining to auditors what happened

Why GKE Enterprise Exists (And Why Google Won't Tell You the Real Reason)

GKE Enterprise exists because basic GKE security sucks on purpose. Google could make clusters secure by default, but they'd rather charge you extra for shit that should be included. Classic enterprise software move.

The enterprise features aren't nice-to-haves—they're the minimum viable security for production:

Workload Identity replaces the JSON key disaster with tokens that actually expire. Instead of long-lived keys floating around your cluster, workloads get short-lived tokens from Google's metadata server. Much better than explaining why API keys are committed to Git.

Binary Authorization stops random containers from reaching production. Only signed, scanned images make it through. I've seen this catch everything from experimental developer containers to actual malware. The deployment delays are annoying until you realize how much crap it blocks.

Policy Controller blocks all those stupid configurations you keep forgetting to prevent. No more pods running as root "just this once." No more missing resource limits that kill your nodes. It's like having someone who actually cares about security review every deployment.

Private clusters fix the stupid "API server on the public internet" problem. Your cluster actually disappears from public access while still working. Exposed API servers are how most attacks start - they're basically a "hack me" sign.

The Real Cost of Kubernetes Breaches

Forget the industry statistics—here's what actually happens when your cluster gets compromised:

When your cluster gets compromised, the timeline is predictable:

First week: Nobody knows what happened or how long attackers have been inside. Everyone panics.

Second week: Expensive security consultants confirm what you already suspected - your RBAC is broken and your secrets management is a joke.

Third week: You rebuild everything from scratch because nobody trusts the existing infrastructure.

Rest of the year: Explaining to compliance teams why customer data was accessible from a compromised development pod.

What Google Won't Tell You

These security features work, but they're not magic. Workload Identity will break half your deployments until you figure out service account binding. Binary Authorization will block emergency releases until you set up proper attestation pipelines. Policy Controller will reject every pod until you write constraints that actually reflect how your applications work.

Implementing this stuff is annoying, but getting fired because your cluster got pwned is worse. Design for compromise from day one.

Workload Identity Architecture

Workload Identity creates a secure bridge between Kubernetes service accounts and Google Cloud IAM service accounts, eliminating those fucking JSON key files that developers commit to Git repos every single time.

How it works: Your pod requests an access token → GKE metadata server exchanges the Kubernetes service account token for a Google Cloud access token → Your app gets temporary credentials without storing keys.

You've decided to actually secure your cluster before it gets pwned. Smart. Here's what I learned from implementing this stuff on real production clusters.

Workload Identity: Getting Rid of JSON Keys

Workload Identity replaces those JSON service account keys that everyone commits to Git repos. The setup is annoying but straightforward once you understand the pieces. Google's Workload Identity documentation covers the complete process. This implements the OIDC token exchange standard for secure authentication.

Enable Workload Identity on your cluster:

gcloud container clusters update your-cluster-name \
    --workload-pool=your-project-id.svc.id.goog \
    --zone=your-zone

Takes about 5 minutes. The cluster needs to restart some components.

Create the service account mapping:

You need a Kubernetes service account and a Google Cloud service account linked together:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: your-k8s-sa
  namespace: production
  annotations:
    iam.gke.io/gcp-service-account: your-gcp-sa@project-id.iam.gserviceaccount.com

Set up IAM binding (this is where most people mess up):

gcloud iam service-accounts add-iam-policy-binding \
    --role roles/iam.workloadIdentityUser \
    --member \"serviceAccount:project-id.svc.id.goog[production/your-k8s-sa]\" \
    your-gcp-sa@project-id.iam.gserviceaccount.com

The bracket format in the member field is crucial. Wrong format means you get permission denied: failed to impersonate service account with zero useful debugging info. I spent 3 hours once debugging this because I missed the damn brackets. Google's error messages are useless.

Common issue: Your app might still try to use mounted JSON keys. Remove any mounted key files or the app will ignore Workload Identity completely. You'll see using Application Default Credentials from JSON file in logs instead of using metadata server credentials.

Binary Authorization: Blocking Unsigned Containers

Binary Authorization prevents deploying unsigned images. Works great once you actually sign your containers.

Flow: Image gets built → Attestor signs the image → GKE checks the policy → If image is signed by trusted attestor, deployment proceeds → Otherwise, deployment fails.

Basic attestor setup:

## Create an attestor
gcloud container binauthz attestors create prod-attestor \
    --attestation-authority-note=projects/PROJECT_ID/notes/prod-note

Policy configuration:

name: projects/PROJECT_ID/policy
defaultAdmissionRule:
  evaluationMode: REQUIRE_ATTESTATION
  enforcementMode: ENFORCED_BLOCK_AND_AUDIT_LOG
  requireAttestationsBy:
  - projects/PROJECT_ID/attestors/prod-attestor
clusterAdmissionRules:
  us-central1-c.your-cluster:
    evaluationMode: ALWAYS_ALLOW  # Remove this after testing
    enforcementMode: ENFORCED_BLOCK_AND_AUDIT_LOG

Start with ALWAYS_ALLOW while you get image signing working. Otherwise you'll block all deployments immediately and spend Friday night figuring out why nothing deploys while your team hates you.

Emergency bypass when you need to deploy unsigned images:

## Temporarily disable enforcement
gcloud container binauthz policy import /path/to/permissive-policy.yaml

## Deploy your fix

## Re-enable enforcement (don't forget this step)
gcloud container binauthz policy import /path/to/strict-policy.yaml

Policy Controller: Preventing Bad Configurations

Policy Controller uses OPA Gatekeeper to enforce security policies. You'll need to write some Rego, but the built-in templates cover most common cases.

Basic constraint to prevent root containers:

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8srequiredsecuritycontext
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredSecurityContext
      validation:
        properties:
          runAsNonRoot:
            type: boolean
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredsecuritycontext
        
        violation[{"msg": msg}] {
            container := input.review.object.spec.containers[_]
            not container.securityContext.runAsNonRoot
            msg := sprintf("Container %v is running as root", [container.name])
        }

Apply the Constraint

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredSecurityContext
metadata:
  name: must-run-as-non-root
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
  parameters:
    runAsNonRoot: true

This blocks containers running as root. Fix your containers first or use warn mode during rollout.

Constraint to require resource limits:

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8srequiredresources
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredResources
      validation:
        properties:
          limits:
            type: array
            items:
              type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredresources
        
        violation[{"msg": msg}] {
            container := input.review.object.spec.containers[_]
            required := input.parameters.limits
            provided := object.get(container, "resources", {})
            limits := object.get(provided, "limits", {})
            
            missing := required[_]
            not limits[missing]
            
            msg := sprintf("Container %v is missing resource limit %v", [container.name, missing])
        }

Network Policies: Controlling Pod-to-Pod Traffic

Network policies are basically iptables rules for pods. They work well once you map out all your service dependencies.

Default behavior: Without network policies, all pods can talk to all pods. With a default-deny policy, nothing can communicate until you explicitly allow it.

Default deny policy (use carefully):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

This blocks all traffic. Apply it in a test namespace first to see what breaks.

Allow DNS resolution (required for most workloads):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to: []
    ports:
    - protocol: UDP
      port: 53
  - to:
    - namespaceSelector:
        matchLabels:
          name: kube-system
    ports:
    - protocol: UDP
      port: 53

DNS resolution breaks without this policy. Every pod needs to resolve service names.

Implementation Reality Check

Enabling these security features will break things. Every implementation I've done follows the same pattern:

• Workload Identity breaks apps expecting JSON keys
• Binary Authorization blocks legitimate deployments
• Policy Controller rejects half your existing workloads
• Network policies break service-to-service communication

Test everything in staging first. Start with permissive policies and tighten them gradually. Have rollback procedures ready before you need them at 3 AM on a weekend.

The security improvements are worth the implementation pain, but plan for several weeks of debugging broken deployments. I spent two months last year slowly rolling out Binary Authorization because every third deployment broke in some creative new way. My incident channel got real quiet after the first few rollbacks.

Security Monitoring That Actually Works

The GKE Security Posture dashboard shows you all the shit that's broken in your clusters. Calling it a "dashboard" is generous - it's more like a fancy to-do list of all the ways you fucked up security.

What you'll see: Lists of pods running as root, containers without resource limits, images with known CVEs, and network policies that don't exist. Basically all the stuff you meant to fix but never got around to.

Beyond the basic stuff, these features actually make your cluster secure instead of just looking secure.

GKE Sandbox: When You Don't Trust Your Own Code

GKE Sandbox (gVisor) gives you an extra kernel layer between containers and your nodes. Sounds great until you realize it breaks half your applications in weird and mysterious ways. The 10-30% performance overhead is real too, and your database team will never shut up about it.

Use cases for gVisor:

• Running untrusted customer code
• Third-party containers with unknown provenance
• Compliance requirements for additional isolation
• Legacy applications with unpredictable system calls

The implementation that works:

apiVersion: v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: gvisor
---
apiVersion: v1
kind: Pod
metadata:
  name: untrusted-workload
spec:
  runtimeClassName: gvisor
  containers:
  - name: app
    image: sketchy/third-party-app
    securityContext:
      runAsNonRoot: true
      runAsUser: 65534  # nobody user
      readOnlyRootFilesystem: true
      allowPrivilegeEscalation: false

gVisor limitations:

• Applications requiring direct filesystem access
• Database drivers using uncommon syscalls
• Performance-sensitive workloads (10-30% overhead)
• Applications with specific volume ownership requirements

Test applications thoroughly with gVisor in staging. Applications that work in regular containers might fail in the sandbox for no obvious reason. Database drivers are particularly bad - they use syscalls that gVisor doesn't support and the error messages make no fucking sense.

External Secrets Operator: Real Secret Management

Kubernetes secrets are just base64-encoded text in etcd. External Secrets Operator pulls real secrets from Google Secret Manager, providing proper rotation and access controls.

The setup that actually works:

## First, create the SecretStore
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: gsm-secret-store
  namespace: production
spec:
  provider:
    gcpsm:
      projectId: \"your-project\"
      auth:
        workloadIdentity:
          clusterLocation: \"us-central1\"
          clusterName: \"prod-cluster\"
          serviceAccountRef:
            name: external-secrets-workload-identity-sa

Then the ExternalSecret that pulls from Secret Manager:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-password
  namespace: production
spec:
  refreshInterval: 15m  # Don't hit API limits
  secretStoreRef:
    name: gsm-secret-store
    kind: SecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
    template:
      type: Opaque
      data:
        password: \"{{ .password | toString }}\"
  data:
  - secretKey: password
    remoteRef:
      key: prod-database-password

Common issues:

• API rate limits: Secret Manager has quotas - don't refresh too frequently
• IAM permissions: Service account needs secretmanager.secretAccessor and proper Workload Identity binding
• Secret rotation: Rotating secrets requires pod restarts - plan your deployment strategy
• Cross-project access: Secrets in different projects complicate IAM configuration

Config Sync: GitOps for Security Policies

Config Sync manages security policies through Git. It works well for fleet-wide configuration, but broken commits can fuck your entire infrastructure. I once pushed a typo in a network policy that broke DNS resolution for 200+ clusters simultaneously. That was a fun Saturday morning.

Repository structure that doesn't suck:

k8s-security-config/
├── fleet/
│   ├── policy-controller/
│   │   ├── templates/
│   │   │   ├── require-non-root.yaml
│   │   │   ├── require-resource-limits.yaml
│   │   │   └── block-privileged.yaml
│   │   └── constraints/
│   │       ├── prod-constraints.yaml
│   │       └── staging-constraints.yaml
│   └── network-policies/
│       ├── default-deny.yaml
│       └── allow-dns.yaml
├── clusters/
│   ├── prod/
│   └── staging/
└── namespaces/
    ├── production/
    └── development/

Config Sync configuration:

apiVersion: configmanagement.gke.io/v1
kind: ConfigManagement
metadata:
  name: config-management
spec:
  git:
    syncRepo: https://github.com/company/k8s-security-config
    syncBranch: main
    secretType: ssh
    policyDir: fleet
  policyController:
    enabled: true
    templateLibraryInstalled: true
    auditIntervalSeconds: 120  # Don't spam the API server

Potential issues:

• YAML syntax errors: Config Sync stops syncing and policies don't update
• Invalid Rego: OPA rejects malformed policies and enforcement stops
• Repository access: Expired SSH keys or changed permissions break synchronization

Emergency rollback procedure:

## Emergency rollback to previous commit
git revert HEAD --no-edit
git push origin main

## Or point to a known-good branch temporarily
kubectl patch configmanagement config-management \
    --type merge \
    --patch '{\"spec\":{\"git\":{\"syncBranch\":\"emergency-rollback\"}}}'

Attack Detection in Practice

Most cluster attacks follow predictable patterns. Focus on detecting common attack vectors rather than complex threat hunting.

Set up alerts that actually matter:

## Failed authentication attempts (brute force attacks)
gcloud logging read '
protoPayload.serviceName=\"container.googleapis.com\"
AND protoPayload.authenticationInfo.principalEmail=\"\"
AND protoPayload.status.code!=0
' --freshness=1h

## Privilege escalation attempts
kubectl get events --field-selector reason=ConstraintViolation \
  | grep -i \"privileged\\|root\\|capabilities\"

## Unusual container deployments
gcloud logging read '
protoPayload.methodName=\"google.container.v1.ClusterManager.CreateNodePool\"
OR protoPayload.methodName=\"io.k8s.core.v1.pods.create\"
' --freshness=10m

High-noise alerts to avoid:

• DNS resolution failures (common in microservices)
• Failed health checks (usually application issues)
• Network policy violations during deployments (often legitimate traffic)

Useful security alerts:

• Multiple authentication failures from same source
• Unsigned container deployments
• Privileged container violations
• API calls from unexpected locations

Set up proper audit logging and actually read it. Most attacks leave obvious traces if you know what to look for.

The Monitoring Setup That Keeps You Sane

Standard Kubernetes monitoring is noisy as hell. Here's what actually matters for security:

Prometheus rules for security events:

groups:
- name: gke-security
  rules:
  - alert: PolicyViolationSpike
    expr: increase(gatekeeper_violations_total[5m]) > 10
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: \"Unusual number of policy violations\"
      description: \"{{ $value }} violations in the last 5 minutes\"

  - alert: BinaryAuthBlocked
    expr: increase(binary_authorization_blocked_total[10m]) > 5
    labels:
      severity: critical
    annotations:
      summary: \"Multiple deployments blocked by Binary Authorization\"

The dashboards that actually help:

1. Policy violations by namespace (find teams that ignore security policies)
2. Binary Authorization blocks by image (identify problematic containers)
3. Workload Identity authentication failures (catch configuration problems)
4. Network policy drops (see what traffic you're blocking)

Incident Response for Compromised Clusters

Even with proper hardening, clusters can get compromised. Have these procedures ready:

Immediate containment steps:

## Isolate the compromised namespace
kubectl annotate namespace compromised-ns \
  networking.policy/isolation=complete

## Scale down suspicious deployments
kubectl scale deployment suspicious-app --replicas=0

## Enable aggressive audit logging
gcloud container clusters update your-cluster \
  --enable-cloud-logging \
  --logging=SYSTEM,WORKLOAD,API_SERVER

Investigation priorities:

• Audit logs for unusual API activity
• Binary Authorization logs for unsigned deployments
• Network policy violations indicating lateral movement
• Workload Identity usage for privilege escalation attempts

Recovery steps:

• Rebuild compromised workloads from verified images
• Rotate potentially compromised secrets
• Strengthen security policies based on attack vector
• Document incident details and prevention measures

Test these procedures regularly. You don't want to learn incident response during an active compromise. Trust me on this one - I spent a very long night figuring out how to isolate compromised pods while attackers were actively moving through the cluster and my phone wouldn't stop buzzing with alerts.