MLOps Production Pipeline: Kubeflow + MLflow + Feast Integration

Everyone says you need "MLOps" but nobody tells you that connecting these tools is like assembling IKEA furniture while blindfolded. Here's the truth about why you might want this setup and what you're getting into.

The Reality Check

Kubeflow handles pipeline orchestration. Setup will make you question your life choices. It's like having a conductor who speaks only in YAML and loses his shit when you miss a comma. The official architecture documentation shows how complex this beast really is. We used 1.8.x but I think there's newer versions now.

MLflow tracks experiments and models. This one actually works pretty decent out of the box, which is why everyone loves it. The model registry saves your ass when you're tracking dozens of model versions. Their tracking server documentation explains the deployment options. Running 2.10-something in our setup.

Feast serves features consistently between training and production. The feature store concept is smart, but deploying it on Kubernetes will make you question your career choices. Read their production deployment guide to understand what you're signing up for.

Why You Actually Want This Integration

Training-Serving Skew Will Destroy Your Career: Your fraud model shows 94% precision in training, then in production it flags every transaction over $5 as fraudulent because someone computed the "time since last transaction" feature differently in the serving pipeline. I've seen this exact bug kill three different models. Feast prevents this nightmare by forcing everyone to use the same feature computation code everywhere.

Experiment Chaos: Without MLflow, you'll have 73 model versions named "final_model_v2_actually_final.pkl" scattered across random S3 buckets. Ask me how I know. The MLflow experiment tracking prevents this nightmare.

Pipeline Reproducibility: Kubeflow ensures that when your model breaks in production (not if, when), you can actually figure out what the hell happened and reproduce the training run. Their pipeline versioning system saves your ass during incidents.

What You Actually Get When It Works

Pipeline Traceability: When shit hits the fan (and it will), you can trace every model back to the exact data, features, and hyperparameters that created it. This has saved my ass more times than I can count.

Consistent Features: Feature computation bugs will make you look like an idiot in front of the business team. Our fraud model went from 95% training accuracy to 60% production accuracy because someone used pandas.rolling() with center=True in training but the serving code used center=False. Took us 3 weeks to find that one line. Feast's offline vs online stores use the same goddamn code, so this shit doesn't happen.

Version Sanity: MLflow keeps track of which model version is actually running where. No more "wait, which model is in production?" panic attacks during incidents. Their model deployment tracking is essential for operations.

Real Production War Stories

Fintech Fraud Detection: Had this running at a payments company. Pipeline was chugging through millions of transactions daily - honestly no idea how many, a lot. Everything's cruising along until one Tuesday morning Kubernetes decides to move our feature extraction job to a different node during peak hours. Fraud scoring just... stopped. For how long? I dunno, felt like forever but probably 20-30 minutes. Business team lost their shit because chargebacks started hitting immediately. Finance was screaming about costs but I never got exact numbers - something like $50K? Maybe more? They don't tell us peons the real damage.

E-commerce Recommendations: Different company, Black Friday disaster. Feast was serving features at decent speed, then 8am EST hits and Redis craps out. Connection timeouts everywhere - ECONNREFUSED 127.0.0.1:6379 spamming our logs. Feature latency went from fast to "users could make coffee while waiting" slow. Everyone got generic recommendations for like 4 hours. Lost conversion was... bad. Really bad. Turned out Redis had shit connection pooling and we maxed out connections. Fixed it with proper pooling config but jesus, who has time to read Redis docs when everything's on fire?

The Learning Curve Reality: Our team of 5 supposedly senior ML engineers took... I think it was 4 months? Maybe closer to 5. Definitely not the "2-4 weeks" bullshit the docs claim. And this was with people who actually knew Kubernetes. If your team is new to K8s, honestly just plan for 6+ months and hope for the best.

Complexity Truth Bomb

Setup: Painful as hell. You need Kubernetes experts, not just data scientists. The Kubeflow installation docs make it look simple but miss tons of gotchas that will wreck your weekend. Check out the troubleshooting guide to see what's coming.

Operations: Plan for 2 full-time DevOps people minimum. These tools break in creative ways that need deep debugging skills. The Kubernetes MLOps patterns documentation helps, but real experience takes time.

Learning Curve: If your team doesn't know Kubernetes well, add 6 months to whatever timeline you're thinking. The CNCF MLOps landscape shows how many moving parts you're dealing with.

The upside? When it works, it actually solves the core MLOps problems. But be realistic about the investment required.

Getting these three tools to actually work together took our team 4 months and way too much coffee. Here's what we learned the hard way so you don't have to.

Infrastructure Prerequisites (Buckle Up)

Kubernetes Cluster Reality:

• Whatever K8s version doesn't crash your other stuff (probably 1.29+)
• Started with 8 nodes, now we have 12 because things kept dying
• 16+ cores per node minimum - learned this when jobs got OOMKilled every Tuesday
• SSD storage or your pipelines will take forever
• RBAC will break in ways that make no sense

Versions We Used (your mileage will vary):

• Kubeflow 1.8.x - newer stuff exists but this worked for us
• MLflow 2.10-something - anything after 2.8 should be fine, earlier versions had issues
• Feast... whatever was stable when we deployed, changes fast
• PostgreSQL 15 - solid choice, 14 had some quirks
• Redis 7.x - don't use ancient versions, they leak memory

Kubeflow Installation (Where Dreams Go to Die)

The Basic Steps:

git clone https://github.com/kubeflow/manifests.git
cd manifests
kubectl apply -k ./manifests/example/

What Will Break (And How):

• RBAC clusterfuck: forbidden: User \"system:serviceaccount:kubeflow:pipeline-runner\" cannot create resource \"pods\" - you'll spend 2 days learning Kubernetes RBAC the hard way
• Istio just stops working. No error messages, no logs. Pods can't talk to each other. curl: (7) Failed to connect to mlflow port 5000: Connection refused
• PVCs stuck forever in Pending because AWS EBS storage class doesn't exist in region us-west-1. Check kubectl get storageclass first, save yourself 3 hours
• UI returns 404 because ingress-nginx is running on port 80 but your ALB is looking for port 8080. The error logs will be useless

MLflow Setup Gotchas:

• Don't use SQLite in production (learned this the hard way when it corrupted) - use PostgreSQL backend
• PostgreSQL needs specific connection pool settings or you'll get timeouts
• S3 artifact store requires EXACT IAM permissions or uploads silently fail
• The tracking server will randomly crash if you don't set memory limits

Feast Integration (Redis Hell)

The Basic Feast Setup:

## Basic feast config - expect this to break
feature_store.yaml: |
  project: mlops_pipeline
  registry: s3://feast-registry/registry.db
  online_store:
    type: redis
    connection_string: redis://redis:6379/0

Feast Pain Points (The Real Shit):

• Redis OOMs at 3am: (error) OOM command not allowed when used memory > 'maxmemory'. Set maxmemory 2gb and maxmemory-policy allkeys-lru or prepare for pain
• DuckDB corrupts itself if you Ctrl+C jobs. database disk image is malformed means you're starting over
• Feature serving goes from 50ms to 2 seconds when Redis memory fragments. redis-cli info memory shows 90% fragmentation - restart Redis, lose data
• Registry corruption: FeatureStore.get_feature_view() returned None means two processes updated registry.db simultaneously. Now your feature definitions are fucked
• CLI version 0.52.0 with server 0.51.x gives you AttributeError: 'FeatureStore' object has no attribute 'get_entity' errors. Version pinning is not optional

Pipeline Integration (Where Everything Breaks)

The Pipeline Code Structure:

## What you think you're building
@components.create_component_from_func
def feature_engineering_component():
    # Load data, compute features, register with Feast
    pass

@components.create_component_from_func  
def model_training_component():
    # Get features from Feast, train model, log to MLflow
    pass

What Actually Goes Wrong (With Exact Errors):

• MLflow tracking URI http://mlflow:5000 doesn't resolve: requests.exceptions.ConnectionError: HTTPConnectionPool(host='mlflow', port=5000): Max retries exceeded. Check your DNS, probably mlflow.kubeflow.svc.cluster.local
• Feast times out: TimeoutError: Redis operation timed out after 5000ms. Redis is choking on feature queries - scale it or simplify your features
• ModuleNotFoundError: No module named 'scikit-learn' because component image has sklearn 1.2 but code expects 1.3. Pin your fucking versions
• OOMKilled: Exit code 137 means memory limit hit. Set resources.limits.memory: 4Gi minimum for any real workload
• dial tcp 10.244.0.15:5000: i/o timeout - network policies blocking pod-to-pod traffic. Check kubectl get networkpolicy
• Schema changes: pydantic.error_wrappers.ValidationError: 1 validation error for GetOnlineFeaturesResponse means someone added a feature without updating the schema

The Reality: Every environment needs different configurations. Hardcode nothing.

Pipeline Orchestration

The High-Level Flow:

@dsl.pipeline(name=\"mlops-pipeline\")
def mlops_pipeline():
    feature_task = extract_features()
    training_task = train_model().after(feature_task)
    deploy_task = deploy_model().after(training_task)

Pipeline Debugging Hell:

• Components fail silently and you have no idea why
• Resource limits are wrong and everything gets killed
• Dependencies between components break in random ways
• Logs are scattered across 15 different places
• Retry logic sometimes makes things worse

Monitoring (Because You'll Need It)

Essential Monitoring Setup:

## Basic prometheus config to not go blind
prometheus.yml: |
  scrape_configs:
  - job_name: 'mlflow'
    static_configs:
    - targets: ['mlflow:5000']
  - job_name: 'feast'  
    static_configs:
    - targets: ['feast:6566']

What We Actually Track:

• Pipeline success rate (ours hovers around 80% on a good day)
• Feature serving latency (spikes when Redis gets cranky)
• MLflow response times (goes to hell during model uploads)
• Kubernetes resource usage (everything uses more than you think)

The Truth: This integration works, but expect to spend half your time debugging infrastructure instead of building models. Get dedicated DevOps help or your data scientists will burn out.

Document everything that breaks, because it'll break again in 6 months when you've forgotten the fix. But when it all works together? You've got a solid MLOps platform that scales and gives you way more control than managed solutions.

Going from "works on my laptop" to "works reliably in production" is a long painful journey, but teams who make it through end up with a real competitive advantage.