Kubeflow - Why You'll Hate This MLOps Platform

Your data scientist runs some version of TensorFlow on their MacBook.

Production runs a different version on Ubuntu with way more RAM. The model works perfectly in the notebook and crashes immediately in production with CUDA_ERROR_OUT_OF_MEMORY. Sound familiar?

That's the bullshit Kubeflow tries to solve

• making your dev environment match production by forcing everything through Kubernetes.

Problem is, now you have ML complexity plus K8s complexity. It's like solving a math problem by adding more math.

What Each Component Actually Does

Kubeflow Pipelines

• DAG runner for ML workflows.

You define steps in Python, it runs them in order. Works most of the time. When it doesn't, you'll grep through logs looking for connection timed out errors because the API server is flaking out again.

Training Operator v1.8.0 (v1.9 broke our distributed training setup)

• Distributed training across GPUs.

Supports Py

Torch, TensorFlow, JAX when the planets align. Here's the gotcha: if you don't set memory limits, one job will OOM the entire node and kill everyone else's work.

Ask me how I fucking know.

Pro tip: Always check kubectl describe pod first.

The events section has the real error, not the useless logs.

KServe

• Model serving that auto-scales when it feels like it.

Way better than writing another Flask wrapper. Handles A/B testing and canary deployments when it's not randomly restarting pods for no reason. The docs miss half the production gotchas but what else is new.

Katib

• Hyperparameter tuning using actual algorithms instead of grid search. Saves you from manually testing 200 learning rates. Uses Bayesian optimization which sounds fancy but really just means "slightly smarter guessing." No idea why this works but don't touch it.

Security Is Your Problem Now

Multi-tenancy means teams can't break each other's stuff. You set up RBAC and Network

Policies, which takes 40+ hours if you want to get it right.

Get it wrong and the intern accidentally deletes prod models.

GPU scheduling is where dreams go to die. Your job shows Pending because the node has a taint you didn't know about. Or the NVIDIA drivers crashed again. Or someone else is hogging all the A100s for their "urgent" hyperparameter search that definitely could have waited.

The Real Decision Tree

Use this if:

• You already run Kubernetes (and someone knows how to debug it)
• Compliance won't let you use cloud services
• You have 2+ DevOps people who don't mind getting paged at 3am
• You need weird custom ML workflows

Run away if:

• You just want to deploy a model (SageMaker exists for a reason)
• Your team is small and wants to focus on actual ML
• You're a startup trying to move fast
• You like having weekends

Current version is 1.8-something which is "stable" in the sense that it breaks in predictable ways. Setup still takes forever and will definitely ruin multiple weekends.

Deployed Kubeflow for 3 different companies. Each time I thought I'd learned from previous mistakes. Each time I discovered new ways for this system to break in production. Here's what really happens.

How Everything Actually Breaks

Kubeflow Notebooks - Jupyter in containers. Great idea until reality hits:

• GPU scheduling fails with cryptic FailedMount errors because someone configured the device plugin wrong
• Persistent volumes corrupt themselves during node restarts (happened 3 times in 6 months - still no idea why)
• Set memory too low: notebook dies loading pandas. Set it too high: OOM kills 4 other people's jobs
• Sharing environments is a nightmare - one person does pip install tensorflow==2.15.0 and breaks everyone's PyTorch models

Actual error from last week: Error: pod has unbound immediate PersistentVolumeClaims. Took 6 hours to figure out the StorageClass was misconfigured.

Central Dashboard - Glorified web portal that fails in creative ways. Auth breaks after every cert-manager rotation. Half the time you'll debug by SSH-ing directly into pods because the UI is lying about what's running.

What Actually Works (Surprisingly)

Distributed Training - Training Operator handles multi-GPU jobs decently when it works. Trained BERT-large across 8x V100s, took 12 hours instead of 3 days on single GPU. GPU utilization hit something like 78% which is respectable for Kubernetes, maybe higher on good days.

Here's the trick: pin everything to versions that actually work together. CUDA whatever, PyTorch whatever, NVIDIA driver whatever-doesn't-crash. Something about NCCL timeouts, I stopped trying to understand.

CUDA 12.1 vs 12.2 driver incompatibility killed an entire week. The error message? Driver/library version mismatch - helpful as always.

Real production example: fraud detection on credit card transactions. Half a billion samples, gradient boosting + deep learning ensemble. Ran distributed across 6 nodes, caught way more fraud than the old system. Saved us a bunch of money, I think the CFO said it was over a million, but also cost us like $400K to run the damn thing.

Pipelines That Don't Suck - Built a manufacturing pipeline that actually works:

1. Pulls IoT sensor data from Kafka (vibration, temperature, oil pressure)
2. Feature engineering with Pandas (took 3 iterations to get memory usage right)
3. Trains anomaly detection models using scikit-learn
4. Deploys via KServe to edge devices

Predicts bearing failures 4 days early. Plant avoided $200K downtime last year, but we also had 2 false alarms that cost $50K in unnecessary maintenance. Still net positive.

LLM Reality Check

Fine-tuning Works - Fine-tuned Llama 2 7B using LoRA on domain-specific support tickets. 4x A100s, 6 hours training time. Used DeepSpeed because the model eats GPU memory for breakfast.

Production gotcha: transformer versions change constantly. Pinned to transformers==4.35.0 after 4.36.0 broke our inference pipeline. Lost 2 days figuring that out while the CEO asked for hourly updates.

Serving Is Expensive - Customer service chatbot on 2x A100s. About 180ms latency for responses, costs $900/day. Still cheaper than hiring more support people but holy shit the GPU bills are brutal.

Model Registry Is Basic - Tracks versions but that's about it. Had to write custom scripts for A/B testing different model versions. The UI looks like it was designed by someone who's never deployed a model.

Storage and Monitoring Hell

Storage - Integrated with S3 and it mostly works. File I/O becomes the bottleneck around 100 concurrent jobs. Had to set up local NVMe caching because network storage is slower than government bureaucracy.

Monitoring - Prometheus + Grafana setup took 2 weeks but now I can see everything. GPU utilization, memory usage, job success rates. Essential for debugging why training jobs randomly die.

The Numbers Nobody Talks About

Measured in actual production:

• GPU utilization: 72-81% (not the magical 90% in vendor benchmarks)
• Training failures: about 15% due to OOM, networking timeouts, or who knows what
• Model serving latency: 120-250ms for 7B models (varies wildly)
• Setup time: at least a month with experienced team, 3+ months if learning K8s from scratch

Scaling bottlenecks:

• Storage IOPS limit hit at around 40 concurrent jobs
• Kubernetes networking falls apart like wet cardboard around 60+ pods doing heavy I/O
• GPU memory fragmentation wastes about 18% capacity on average

Works once it's running but getting there nearly broke my sanity. Actually, this part worked better than I expected. Still, budget 6 months and hire a therapist.