Stop Your ML Pipelines From Breaking at 2 AM

Setting up Kubeflow and Feast in production isn't like following a fucking cookbook. It's like trying to assemble IKEA furniture while the instructions are on fire and your manager is asking when it'll be ready every 10 minutes.

Why This Guide Won't Bullshit You

I spent the better part of 2024 getting this stack to work properly. The official docs assume you're a Kubernetes wizard who never makes mistakes. Real talk: you're going to break things, and that's fine.

Here's what actually happens when you try to run ML in production:

• Your notebook that "totally works" will fail spectacularly when it tries to load 50GB of training data
• Kubeflow will eat all your memory and ask for more
• Feature serving will randomly return stale data and you won't notice until a model starts predicting that every customer wants to buy pet insurance
• The pipeline that worked fine for 3 weeks will suddenly decide to crash at 2 AM on a Sunday

What You're Actually Building

A system that can handle your ML team's chaos without requiring a full-time babysitter:

Infrastructure That Doesn't Suck:

• Recent Kubeflow that won't randomly break (we're running something recent, check what's current)
• Feast feature store that actually keeps your features consistent
• A Kubernetes cluster that can survive your data scientist's massive training jobs
• Storage that won't randomly delete your models (this has happened to me twice)

Pipeline Magic:

• Model serving that doesn't time out when someone hits refresh
• Feature engineering that handles time zones correctly (seriously, fuck time zones)
• Model versioning that lets you roll back when the new model decides cats are vegetables
• Monitoring that actually tells you useful shit when things break

Production Reality:

• Authentication that doesn't make everyone an admin by default
• Resource limits so one person can't crash the entire cluster
• Backups that you'll pray you never need but will save your ass

Time Expectations (AKA The Truth)

• Initial setup: Plan for a full weekend. The "quick start" guides are lying.
• Actually working system: Add another week for all the edge cases the docs don't mention
• Production ready: At least a month before you'd trust this with real business data
• Team onboarding: Your data scientists will need hand-holding for at least 2 weeks

The Infrastructure Tax

You'll need more resources than you think:

Minimum viable cluster:

• 3 nodes with decent CPUs and lots of RAM (think 16 cores, 64GB-ish if you can afford it)
• Fast storage - like 500GB+ of NVMe if you don't want to wait forever
• Decent network between nodes (don't cheap out here)

Reality check:

• Your cluster will use a huge chunk of resources just sitting there doing nothing
• ML training jobs are memory hogs that will OOM kill everything in sight
• Feature stores need fast storage or your response times go to shit

What Actually Breaks in Production

"Container won't start"

• Docker images that work on your laptop but fail in production
• Memory limits set too low (learned this the hard way)
• Missing environment variables that worked fine in development

"Features are inconsistent"

• Clock drift between systems causing feature freshness issues
• Race conditions during feature materialization
• Different Python versions computing features slightly differently

"Everything is slow"

• Network latency you didn't account for
• Database connections not properly pooled
• Images being pulled from slow registries every time

"It worked yesterday"

• Kubernetes node ran out of disk space
• Certificate expired (always happens at night)
• Someone changed a config and didn't tell anyone

This guide will walk you through the actual solutions to these problems, not just the happy path that works in demos.

The Essentials (stuff I actually used):

• Kubeflow docs - where you'll spend most of your debugging time
• This Stack Overflow thread - more useful than the official docs
• Feast deployment guide - only one that actually worked for me

Look, this is going to hurt. Plan for 6-8 hours minimum, and that's if everything goes smoothly (narrator: it won't).

Step 1: Setting Up Your Cluster (Plan for 2-4 Hours Because Something Will Break)

First things first - you need a Kubernetes cluster that won't fall over when your data scientist decides to train a model on the entire internet.

Resource requirements that won't leave you crying:

• 3 nodes minimum (because one will always be "upgrading" at the worst possible time)
• Decent CPUs and lots of RAM per node - like 16 cores and 64GB if you can swing it (ML workloads are hungry beasts)
• Fast storage - at least 500GB of NVMe if you don't want to wait forever for models to load

## This will fail if your username has spaces (learned the hard way)
eksctl create cluster \
  --name kubeflow-prod \
  --version 1.31 \
  --region us-west-2 \
  --nodegroup-name workers \
  --node-type m5.4xlarge \
  --nodes 5 \
  --nodes-min 3 \
  --nodes-max 10

## This command will sit here for 15 minutes, go get coffee
kubectl get nodes

Pro tip: AWS will happily create your cluster in a region where you have no other resources. Check your region twice before hitting enter, or you'll be debugging cross-region network issues for days.

Step 2: Storage That Won't Randomly Delete Your Models

I lost 3 days of training data once because I used the wrong storage class. Don't be me.

## This creates storage that actually persists
kubectl apply -f - <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: kubeflow-fast
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp3
  iops: "3000"  # You'll need this when 50 pods try to read the same model
reclaimPolicy: Retain  # DO NOT use Delete unless you enjoy data loss
allowVolumeExpansion: true
EOF

Step 3: Installing Kubeflow (Abandon Hope, All Ye Who Enter Here)

The documentation says this takes 30 minutes. It's lying. Budget 2-3 hours, and that's if you're lucky.

## Clone the repo (this actually works)
git clone https://github.com/kubeflow/manifests.git
cd manifests
git checkout v1.10-branch

## This command looks simple but will break in 17 different ways
kubectl apply -k ./manifests/v1.10/

What will definitely go wrong:

• Some random CRD called destinationrule.v1beta1.networking.istio.io won't install because Kubernetes 1.31 deprecated shit that worked fine in 1.30
• Istio will OOM kill itself and take down half your cluster because someone set the memory limit to 512Mi (spoiler: it needs like 2GB)
• MySQL pod throws Error 1 (HY000): Can't create/write to file '/tmp/#sql_5_0.MYD' because you forgot to mount persistent storage and it ran out of space after 10 minutes
• Everything sits in "Pending" status while you stare at it for an hour before realizing you never applied the fucking storage class

The fix that actually works:

## Install components one at a time so you can debug failures
kubectl apply -k ./manifests/common/
kubectl wait --for=condition=Ready pods --all -n kubeflow --timeout=600s

## If something is stuck, delete it and try again
kubectl delete pod <stuck-pod> -n kubeflow

Step 4: Feast Setup (Where Dreams Go to Die)

Feast documentation assumes you understand every storage system ever created. You probably don't, and that's fine.

## Create namespace first or nothing works
kubectl create namespace feast-system

## Install Feast with sane defaults
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: feast-server
  namespace: feast-system
spec:
  replicas: 1  # Start with 1, scale later when it's actually working
  selector:
    matchLabels:
      app: feast-server
  template:
    metadata:
      labels:
        app: feast-server
    spec:
      containers:
      - name: feast-server
        image: feastdev/feature-server:0.53.0
        env:
        - name: FEAST_ONLINE_STORE_TYPE
          value: "redis"
        resources:
          requests:
            memory: "1Gi"  # Start small
            cpu: "500m"
          limits:
            memory: "2Gi"  # This will probably OOM anyway
            cpu: "1"
EOF

Redis setup that won't make you hate life:

## Use a managed Redis if you can afford it
## Self-hosted Redis will eat your storage and ask for more
helm repo add bitnami https://charts.bitnami.com/
helm install redis bitnami/redis \
  --namespace feast-system \
  --set auth.enabled=false \
  --set master.persistence.size=50Gi

Step 5: Testing Your Frankenstein's Monster

Here's how you know if it's actually working:

## Check if basic shit is running
kubectl get pods -n kubeflow | grep -v Running
## If this returns anything, start debugging

## Port forward to access the UI (because ingress is hard)
kubectl port-forward -n kubeflow svc/ml-pipeline-ui 8080:80
## Access the UI (should show the Kubeflow Pipelines interface)

Load test with actual traffic:

## This will tell you if your cluster can handle more than one user
for i in {1..10}; do
  kubectl run load-test-$i --image=curlimages/curl --rm -it --restart=Never -- \
    curl -X GET http://ml-pipeline-ui.kubeflow:80/apis/v1beta1/pipelines \
    -H "Accept: application/json"
done

Step 6: Monitoring (Because You'll Need It)

Don't skip this. When things break at 3 AM (and they will), you'll want to know why.

## Prometheus that actually works
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --set grafana.adminPassword="change-this-password"

## Get the Grafana URL
kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80
## Login with admin/change-this-password

Step 7: Backups (Because You Don't Want to Do This Again)

## Backup everything important (trust me on this)
kubectl get all -n kubeflow -o yaml > kubeflow-backup-$(date +%Y%m%d).yaml
kubectl get all -n feast-system -o yaml > feast-backup-$(date +%Y%m%d).yaml

## Store these somewhere safe, like NOT on the same cluster

Time reality check:

• Initial setup: Plan for most of a weekend, maybe longer if you hit weird issues
• First working pipeline: Add another week, maybe more
• Production ready: Weeks of constant tweaking and fixing edge cases
• Actually stable: Months of running it with real workloads

This setup works. I know because I've broken it in every possible way and fixed it again. Your mileage may vary, but at least you'll know what to expect when it inevitably catches fire.

When You're Really Stuck:

• Kubeflow manifests repo - the actual installation files you'll be debugging
• This AWS EKS guide - at least it works
• Kubeflow Slack - real humans who've felt your pain

Congratulations, you got it running. Now comes the fun part: keeping it running when real users start hitting it.

Resource Management (Or: How I Learned to Stop Worrying and Love OOMKilled)

Memory Limits Are Lies

The demo tutorials use tiny datasets. Real ML training will eat your RAM for breakfast and ask for seconds. I learned this when our recommendation model ate way more memory than expected - like 40-something GB - and crashed a bunch of other stuff running on the same nodes.

## Resource limits that sort of work (your mileage will vary)
apiVersion: v1
kind: LimitRange
metadata:
  name: maybe-realistic-limits
  namespace: ml-production
spec:
  limits:
  - type: Container
    default:
      cpu: "1"  # Start here, adjust when shit breaks
      memory: "2Gi"  # This will probably be wrong
    defaultRequest:
      cpu: "500m" 
      memory: "1Gi"  # Also probably wrong
    max:
      cpu: "64"  # Someone always needs way more than expected
      memory: "256Gi"  # Yes, really
    min:
      cpu: "50m"  # Even tiny things need something
      memory: "64Mi"

GPU Scheduling (AKA Expensive Disappointment)

GPUs are expensive and everyone wants them. I watched our team burn through $3,000 in V100 hours in one weekend because someone left a hyperparameter search running that spawned 200 jobs. Your data scientists will fight over them like seagulls over french fries, and then leave them idle running Jupyter notebooks "just in case."

What actually works:

• Set strict time limits on GPU jobs (4 hours max, fight me)
• Use node taints to keep GPU nodes for actual GPU workloads
• Monitor GPU utilization religiously (someone is always mining crypto)
• Have a queue system or people will submit 47 jobs at once

Storage and Performance Reality

Fast Storage Costs Money, Slow Storage Costs Sanity

Feature serving usually takes like 150ms on a good day, sometimes 300-400ms when AWS decides to route traffic through Mars. I've seen it hit 2 seconds when Redis decides to do a background save at the worst possible moment. Invest in NVMe storage or your 50ms SLA turns into "maybe sometime today."

Monitoring (Because Everything Will Break)

The Metrics That Actually Matter

Forget vanity metrics. Monitor these or you'll be debugging blind:

• How many pipelines failed in the last hour (not success rate)
• Feature serving latency at the 99th percentile (averages lie)
• Whether your feature store data is actually fresh
• Memory usage across all nodes (someone will always hit the limit)
• Disk space (you'll run out at 3 AM on Sunday)

Security (Don't Skip This)

Everyone thinks they'll add security later. Later never comes, then you get breached.

Basic shit that works:

• Use network policies to limit pod-to-pod communication
• Store secrets in Kubernetes secrets, not environment variables
• Enable RBAC and don't give everyone cluster-admin
• Rotate your keys occasionally

Backups (You'll Thank Me Later)

## Simple backup that actually works
kubectl get all -n kubeflow -o yaml > kubeflow-backup-$(date +%Y%m%d).yaml
kubectl get all -n feast-system -o yaml > feast-backup-$(date +%Y%m%d).yaml

## Store these off-cluster, preferably in another cloud

Performance Tips From The Trenches

Pipeline Performance:

• Run independent steps in parallel (duh)
• Cache expensive feature computations
• Use smaller Docker images (saves 2-3 minutes per job)
• Pre-pull images on nodes

Feature Store Performance:

• Redis clustering for high availability
• Use connection pooling
• Monitor cache hit rates
• Set TTLs on features (stale data is worse than no data)

Cost Control (Before Your CFO Kills You)

• Set up auto-scaling (scale to zero at night)
• Delete old pipeline runs automatically
• Use spot instances for training jobs
• Monitor your bill religiously (cloud costs compound daily)

The key insight: production is about preventing problems, not just solving them. Plan for failure because it's coming whether you're ready or not.

Actually Useful Stuff:

• This K8s resource guide - saved me hours debugging OOM kills
• Prometheus monitoring basics - how to actually query your metrics
• Feast production tips - wish I'd found this earlier