Deploy Feast in Production Without Losing Your Mind

I've been running Feast in production since 0.47 and let me tell you - it was a fucking nightmare until recently. The recent 0.53.x versions have been way more stable than the 0.52.x shitshow. We finally stopped getting silent materialization failures that cost us 2 weeks of debugging and a very angry VP of Engineering.

What Actually Changed in 2025

Look, Feast went from "experimental toy that breaks constantly" to "production infrastructure that only breaks occasionally." Here's what happened:

Canonical Charmed Feast dropped on July 10, 2025 and it's basically "Feast but someone else deals with the 3am alerts." If you can afford enterprise support (probably more than their initial $100k+/year estimate once they see your actual usage), worth investigating. Ubuntu people know how to package software properly.

Recent versions actually work: The 0.53.x releases fixed a bunch of shit that made 0.52.x unusable:

• Silent materialization failures finally scream at you instead of eating your data
• Memory leaks that killed our weekend deployments seem to be fixed (knock on wood)
• Connection pooling doesn't completely shit itself under load anymore
• You get actual Prometheus metrics instead of guessing why things are slow

I upgraded from 0.52.2 and didn't lose data for the first time in 6 months. Could be luck, but I'll take it.

The Vector Search Thing

In March 2025 they added alpha vector search support for RAG applications. It's alpha quality so don't put it in production yet, but the idea is solid - combine your feature store with vector similarity search so you don't need separate systems.

The Milvus integration works for document retrieval if you have under 100M vectors. Above that, you're back to managing separate systems anyway. For production vector search, stick with Pinecone, Weaviate, or Qdrant until Feast's integration matures. The Feast roadmap shows they're working on better vector database support, but it'll be months before it'll be production-ready.

Performance Improvements That Matter

Dragonfly replaced Redis in our setup and holy shit the difference is real. Redis was choking around 50k ops/sec. Dragonfly handles way more - I've seen it do 300k+ ops/sec on the same hardware, sometimes more depending on the workload. Your mileage will vary but it's night and day. It's Redis-compatible so you just change the connection string and suddenly your feature serving doesn't suck. Check the Dragonfly performance benchmarks and Redis comparison results for detailed numbers.

DuckDB for offline stores makes sense if your historical data is under 10TB. We saved a shit-ton of money switching from BigQuery to DuckDB. I think it was like 8-12k per month? Maybe more during heavy query months. Queries are actually faster and setup takes an hour instead of configuring IAM hell. The DuckDB integration guide walks through the setup, and performance comparisons show why it beats traditional data warehouses for smaller datasets. Consider MotherDuck for managed DuckDB if you want the performance without the ops overhead.

The Three Deployment Patterns That Work

Cloud-Native (What Everyone Does):

• Kubernetes + BigQuery/Snowflake + Redis/DynamoDB
• Costs $15k-30k/month at scale but battle-tested
• Good if you already have cloud ops teams and existing Kubernetes infrastructure

Enterprise Supported (New in 2025):

• Charmed Feast + on-premises or cloud
• $100k+/year but includes enterprise support and SLAs
• Makes sense if compliance matters more than money and you need 24/7 support

High-Performance (For When Latency Matters):

• DuckDB + Dragonfly + bare metal or cloud VMs
• Sub-millisecond P99 latency, costs 50% less than cloud
• Requires actual infrastructure engineering and performance tuning expertise

After running Feast in production for 18 months across 3 different deployments, here's what actually works and what will make you want to quit your job.

Kubernetes: Less Painful Than DIY

The Feast Operator moved from "experimental garbage" to "actually usable" in 2024. It's still alpha but at least it doesn't randomly delete your data anymore. I learned this the hard way - version 0.48.x deleted my entire Redis cluster during a routine upgrade. Fun times explaining that to the product team. Check the Kubernetes deployment guide and Helm charts for production setup. The operator documentation shows all available configuration options.

## This actually works now (mostly) - test everything in staging first
apiVersion: feast.dev/v1alpha1
kind: FeastStore
metadata:
  name: production-feast
spec:
  offlineStore:
    type: bigquery
    project: your-ml-project  # don't use "ml-platform-prod" like everyone else
  onlineStore:
    type: redis
    replicas: 3
    memoryLimit: 16Gi  # Start with 8Gi, you'll need more
  featureServer:
    replicas: 5  # 2 replicas if you hate availability
    resources:
      cpu: 2
      memory: 4Gi  # Memory leaks still happen - restart jobs every 24 hours or watch containers OOM at 3am

Reality check: The operator handles basic scaling but you'll still be writing custom monitoring and debugging deployment issues. It's better than raw YAML hell but don't expect magic.

Performance: Dragonfly Saved Our Asses

Redis hits a wall around 50k-100k ops/sec in production depending on your key sizes. We tried scaling horizontally and it was a clusterfuck of connection pooling issues and hot key problems. Spent 2 weeks debugging why some feature requests took 500ms when others took 2ms - turns out one Redis node was getting hammered while others sat idle.

Dragonfly is Redis-compatible and handles 10x more load on the same hardware. Migration took one afternoon:

## Literally just change the connection string
export FEAST_ONLINE_STORE_CONNECTION_STRING="dragonfly-cluster.internal:6379"
## Test with small traffic first, obviously

Gotchas: Dragonfly uses more memory per key but way less CPU. Budget accordingly. Some Redis-specific Lua scripts might break but Feast doesn't use the weird ones.

Cost Optimization That Worked

DuckDB for Offline Store: We saved $12k/month switching from BigQuery to DuckDB for our 4TB historical dataset. Queries are faster and no surprise bills from rogue analytical queries.

Right-size your shit:

• Feast servers: 2 CPU/4GB minimum, scale from there based on actual load
• Redis memory: 3x your feature data size (overhead is real)
• Connection pooling: 50-100 connections per Feast server, tune based on latency

Materialization scheduling: Run big materialization jobs at 3AM when AWS/GCP charges 60% less. Set up proper alerts so you know when they fail.

Security: Do This or Get Fired

🔒

Network isolation: Private VPC, no public IPs, VPN or bastion for access. Basic stuff but people fuck this up constantly. I've seen prod Feast instances with Redis open to the internet. Don't be that person. Follow the VPC security best practices and use network policies in Kubernetes.

Encryption everywhere:

• TLS for Redis with SSL (costs 5% performance, worth it)
• Customer-managed keys for BigQuery (compliance team will love you)
• Encrypt backups (duh)

Access control:

• Different service accounts for different environments
• API key rotation every 90 days (automate this or you'll forget and get locked out at the worst possible time)
• Audit logs for everything (recent Feast versions finally have decent logging)
• Use RBAC policies and Pod Security Standards

What Still Breaks

Vector search is alpha: Don't use it yet. It's alpha quality and the Milvus integration breaks under load - trust me, I tested it. Our vector similarity queries worked fine with 1000 documents but completely shit the bed at 100k. Wait 6-12 months or stick with dedicated vector databases.

Upgrades are scary: Plan for downtime. Test in staging with real data. Have rollback procedures. Feast doesn't guarantee backward compatibility and I've seen minor version upgrades break existing feature definitions. Budget 2-4 weeks for major version upgrades if you have complex schemas.

Memory leaks still exist: Not as bad as 0.52.x but long-running materialization jobs still leak memory. Restart them every 24 hours or you'll wake up to OOMKilled containers at 3am.

Connection pooling: Gets weird under high load. The default connection pool size is 10 which is completely useless in production. Start with 100. Monitor connection counts and set aggressive timeouts - I've seen hanging connections eat all available Redis connections and bring down the entire feature serving.

Monitoring You Actually Need

📊

## Essential metrics to alert on
feast_materialization_job_failures_total  # Page immediately
feast_serving_latency_p99_seconds > 0.1   # Warn after 5 minutes  
redis_memory_usage_percentage > 80        # Scale or clean up data
feast_feature_freshness_hours > 4         # Features getting stale

Don't monitor everything - you'll get alert fatigue. Focus on materialization failures, serving latency, and memory usage.