Kubeflow & Feast Production MLOps Setup: AI-Optimized Technical Reference

Critical Configuration Intelligence

Resource Requirements (Real-World Minimums)

• Minimum viable cluster: 3 nodes, 16 cores/64GB RAM per node, 500GB+ NVMe storage
• Reality check: Cluster uses significant resources at idle; ML training jobs are memory-intensive and will OOM kill other processes
• GPU considerations: Set 4-hour max time limits; use node taints for GPU-only workloads; monitor utilization (cryptocurrency mining risk)

Time Investment Reality

• Initial setup: Full weekend minimum (documentation claims 30 minutes are false)
• Actually working system: Additional week for undocumented edge cases
• Production ready: Minimum 1 month before trusting with business data
• Team onboarding: 2 weeks minimum hand-holding required

Storage Configuration That Prevents Data Loss

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: kubeflow-fast
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp3
  iops: "3000"
reclaimPolicy: Retain  # CRITICAL: DO NOT use Delete
allowVolumeExpansion: true

Critical Warning: Using Delete reclaim policy will cause data loss. Author lost 3 days of training data from this mistake.

Installation Failure Modes & Solutions

Common Kubeflow Installation Failures

1. CRD compatibility issues: Kubernetes 1.31 deprecated APIs that worked in 1.30
2. Istio OOM kills: Default 512Mi memory limit insufficient, needs 2GB minimum
3. MySQL storage failures: Error 1 (HY000) indicates missing persistent storage mount
4. Pending pod status: Usually missing storage class application

Working Installation Pattern:

# Install components sequentially for debugging
kubectl apply -k ./manifests/common/
kubectl wait --for=condition=Ready pods --all -n kubeflow --timeout=600s

# Delete and retry stuck pods
kubectl delete pod <stuck-pod> -n kubeflow

Feast Setup Reality

• Documentation assumes expertise in all storage systems
• Redis setup requires managed service or significant storage allocation
• Default memory limits (1-2Gi) insufficient for production workloads

Production Performance Characteristics

Feature Serving Latency Expectations

• Good day: 150ms typical response time
• Bad day: 300-400ms when AWS routes through suboptimal paths
• Disaster: 2+ seconds during Redis background saves
• Solution: Invest in NVMe storage or accept degraded 50ms SLA performance

Memory Consumption Patterns

• Demo tutorials: Use tiny datasets, unrealistic for production
• Reality: Recommendation models consume 40GB+ RAM, crash co-located services
• Resource allocation strategy: Memory requests at 75% of estimated need, limits at 150% of requests

Cost Analysis & Alternatives Comparison

Platform	Setup Difficulty	Monthly Cost	Reliability	Kubernetes Integration
Kubeflow + Feast	Excruciating (weeks)	$8K-25K	Eventually works after fixes	Native
AWS SageMaker	Annoying (days)	$20K-80K+ (surprise bills)	Usually works on happy path	Limited
Google Vertex AI	Reasonable (hours)	$15K-50K (TPU costs)	Actually tested by Google	Partial
Azure ML	Tolerable (days)	$12K-40K (hidden costs)	Sometimes works	Half-implemented
MLflow + DIY	Months of pain	$3K + sanity cost	Define "work"	You implement it

Critical Monitoring Requirements

Metrics That Prevent 3AM Failures

• Pipeline failure count (last hour) - not success rates
• Feature serving latency at 99th percentile - averages mislead
• Feature store data freshness verification
• Memory usage across all nodes - someone always hits limits
• Disk space monitoring - failures occur at 3AM Sunday

Storage Growth Patterns

• Getting started: 500GB lasts few weeks
• Small team: 2-5TB realistic need
• Bigger team: 10-20TB with growth management
• Enterprise: 50TB+ requires cleanup automation

Critical: Storage grows rapidly from model artifacts, training data, and pipeline intermediates. Implement automated cleanup or face sudden capacity failures.

Security Implementation (Non-Optional)

• Network policies for pod-to-pod communication restriction
• Kubernetes secrets for credential storage (not environment variables)
• RBAC enablement without universal cluster-admin access
• Regular key rotation implementation

Backup Strategy (Data Loss Prevention)

# Simple backup that works
kubectl get all -n kubeflow -o yaml > kubeflow-backup-$(date +%Y%m%d).yaml
kubectl get all -n feast-system -o yaml > feast-backup-$(date +%Y%m%d).yaml

Critical: Store backups off-cluster, preferably different cloud provider.

Common Production Failure Scenarios

Feature Inconsistency Between Training/Serving

• Root causes: Clock drift between systems, race conditions during materialization, Python version differences in computation
• Detection: Compare online vs offline store values for same entities
• Resolution: Synchronize system clocks, implement atomic feature updates, standardize computation environments

Pipeline Performance Degradation

• Symptoms: Slow execution, resource starvation
• Common causes: I/O bottlenecks from slow storage, network bandwidth limits, inefficient data loading, excessive small pipeline steps
• Solutions: Profile pipeline code, rightsize resource requests, combine related operations

System Unresponsiveness Recovery

1. Check cluster health: kubectl get nodes
2. Identify failing components: kubectl get pods -n kubeflow | grep -v Running
3. Check resource exhaustion: kubectl describe nodes | grep -A 5 "Allocated resources"
4. Restart services in dependency order: API server first, then UI

Resource Scaling Guidelines

Concurrent Pipeline Capacity

• Small cluster: 5-10 simple pipelines maximum
• Decent cluster: 20-50 concurrent pipelines (resource dependent)
• Large cluster: 100+ possible with adequate hardware

Limiting factors: Resource availability and pipeline complexity, not just count.

Feature Store Production Considerations

Redis Configuration for High Availability

• Use clustering for availability
• Implement connection pooling
• Monitor cache hit rates
• Set appropriate TTLs (stale data worse than no data)

Feature Freshness Monitoring

# Alert when features > 1 hour stale
feature_freshness_gauge = Gauge('feast_feature_freshness_seconds',
                               'Feature freshness in seconds', ['feature_view'])

Database Recovery Procedures

Corruption Recovery Steps

1. Stop pipeline services: kubectl scale deployment ml-pipeline-api-server --replicas=0
2. Check integrity: CHECK TABLE pipeline_runs;
3. Restore from backup if corrupted
4. Restart services after verification

Critical: Maintain automated daily database backups to minimize data loss.

Cost Control Strategies

• Auto-scaling with scale-to-zero at night
• Automated deletion of old pipeline runs
• Spot instances for training jobs
• Daily bill monitoring (costs compound rapidly)

Performance Optimization Patterns

• Parallel execution of independent pipeline steps
• Expensive feature computation caching
• Smaller Docker images (saves 2-3 minutes per job)
• Pre-pulling images on nodes

Troubleshooting Decision Tree

ImagePullBackOff Errors

1. Check node internet connectivity
2. Verify resource constraints
3. Examine network policies for image registry access
   Common cause: Nodes too small or network policy blocking downloads

OOMKilled Containers

1. Check memory usage patterns in logs
2. Monitor container resource consumption
3. Adjust resource limits (start with 2x current allocation)
   Pattern: Set memory requests to 75% estimated need, limits to 150% of requests

Service Communication Failures

1. Test DNS resolution between services
2. Check service endpoints configuration
3. Verify network policies allow cross-namespace traffic
   Common issue: DNS problems or overly restrictive network policies

This technical reference extracts all actionable intelligence while preserving the operational context that prevents common implementation failures. The original author's hard-won experience provides critical guidance for avoiding expensive mistakes and lengthy debugging cycles.