Kafka + MongoDB + Kubernetes + Prometheus Integration - When Event Streams Break

Your traditional APM tools are completely useless when it comes to event-driven architectures. When a user clicks "submit order" and that triggers a dozen microservices through Kafka topics, and one of them dies 30 seconds later, good luck figuring out which one shit the bed.

I learned this lesson after we spent about 4 hours debugging an order processing failure on a Tuesday night, only to discover that the payment service was dying because MongoDB connection pooling was set to some ridiculously low default - I think it was like 10 connections or something. The user got charged, but their order vanished. Try explaining that disaster to the business team in the morning.

The Real Problems Nobody Talks About

Async Hell: When everything is asynchronous, debugging becomes pure guesswork. Your logs are scattered across services with different timestamps - some in UTC, some in EST because someone forgot to fix that one service - and maybe correlation IDs if you're lucky. I watched a team waste 3 days trying to trace a single failed event through their system because nobody bothered implementing distributed tracing properly.

Kafka's "At Least Once" Bullshit: Kafka's "at least once" delivery is code for "we'll duplicate your shit and you deal with it." We had customers getting charged twice for the same order because duplicate payment events went through and our monitoring was too stupid to catch it. Only found out when customer support started getting death threats via email.

State Synchronization Nightmare: Each service has its own MongoDB database, and keeping track of global state across services is like herding cats. When the user service thinks an account is active but the billing service thinks it's suspended, who's right? Your monitoring better have an answer. This is where event sourcing patterns and CQRS can help, but they introduce their own complexity.

The Late-Night Debugging Reality: Picture this - your phone buzzes because users can't buy anything. You stumble to your laptop, log into your Kubernetes dashboard, see that all the pods are green, check Prometheus and see normal CPU/memory usage. Everything looks fine, but orders are still failing and people are starting to ask questions. This is exactly why you need actual event-driven observability, not just pretty green circles in Grafana.

Distributed Debugging Challenge: Unlike monoliths where you can step through code, event-driven systems make you trace events across multiple services, databases, and message queues to figure out what the hell happened. Tools like Jaeger and Zipkin help with this, but you need to instrument everything properly or they're useless.

Why This Stack Actually Works (When Configured Right)

After breaking production way too many times, I finally figured out that Kafka + MongoDB + Kubernetes + Prometheus actually gives you the visibility you need to keep your job. But every vendor conveniently forgets to mention all the gotchas that will screw you.

Kafka Event Tracing: Kafka 2.8+ includes trace headers that let you track events across the entire system. But here's the part that will make you want to quit - you have to manually propagate correlation IDs through every single service in your event chain. Miss ONE fucking service and your beautiful tracing becomes useless. OpenTracing or OpenTelemetry can automate this, but plan on spending a week debugging why service #17 isn't passing headers correctly. The W3C Trace Context specification helps standardize this across different tools.

MongoDB Change Streams for State Tracking: MongoDB Change Streams can publish database changes as events, which is brilliant for keeping track of state changes. But here's where MongoDB will bite you in the ass - change streams will absolutely flood your Kafka cluster if you don't filter them properly. We accidentally pushed... fuck, I think it was like 70-something GB of change events per day? Maybe 80GB? Could have been more - I stopped looking at the disk usage graph after our storage costs hit $400 that week and the ops team started asking uncomfortable questions.

Prometheus for Everything Else: Prometheus excels at collecting metrics from all these components. The JMX exporter gives you Kafka broker metrics, MongoDB exporter handles database metrics, and kube-state-metrics covers Kubernetes. But Prometheus will eat your entire disk if you're not careful with retention - some asshole on our team added user IDs as metric labels and murdered our entire monitoring stack. I think it was consuming like 180GB? Maybe 220GB? The disk filled up so fast I couldn't even check the exact number before everything crashed. That was a fun 2am conversation with the infrastructure team.

Kubernetes Service Discovery: Kubernetes service discovery automatically configures Prometheus to scrape new services. This works great until you have 200 microservices and Prometheus starts choking on all the scraping. Prometheus recording rules become mandatory or your dashboards will timeout and make you look like an idiot in front of the whole team.

The Cost Reality Check

Production Monitoring Reality: A typical Kafka monitoring dashboard shows broker health, consumer lag, throughput rates, and error counts across dozens of topics and consumer groups - essential visibility that standard APM tools miss completely.

Before you get a hard-on for this stack, let me crush your dreams with the actual cost:

• Kafka: Expect $500-1500/month for a production 3-broker cluster on AWS, depending on throughput
• MongoDB: 3-node replica set with 32GB RAM each = $800-2000/month
• Prometheus: 16GB RAM minimum for serious workloads, 32GB if you want decent retention
• Kubernetes: Add 30-50% overhead for pod scheduling, networking, and monitoring

Total monthly cost for a medium-scale deployment: somewhere between $2000-5000/month, though our last AWS bill was closer to $6200 because of some egress charges we didn't see coming. Your manager will shit themselves when they see this bill and start asking what the hell you're spending money on.

Version Compatibility Hell

Here's what actually works together as of 2025-09-04:

• Kafka 3.5.x: Stable, avoid 3.6.0 - consumer group stability issues will crash your brokers under load
• MongoDB 7.0.x: Works well, but the Percona MongoDB exporter 0.40.0 breaks with auth enabled - use 0.39.0
• Kubernetes 1.28+: Required for recent Prometheus operator features
• Prometheus 2.47+: Needed for native histogram support

Don't mix and match randomly - these specific versions have been battle-tested in production environments.

What Nobody Tells You About Implementation

Start Small: Don't try to monitor everything day one. Begin with basic Kafka consumer lag and MongoDB connection counts. Add complexity gradually or you'll drown in metrics noise. The Four Golden Signals from Google SRE are a good framework: latency, traffic, errors, and saturation.

Correlation IDs are King: Every event needs a correlation ID that flows through your entire system. Generate them at the API gateway and log them everywhere. This single change will save you hours of debugging.

Alert Fatigue is Real: We started with 50+ alerts and got pages every hour. Now we have 5 critical alerts that actually matter: Kafka consumer lag > 10k messages, MongoDB replica set degraded, any service error rate > 5%, Kubernetes node not ready, Prometheus server OOM. Everything else is a dashboard metric.

Resource Planning: Prometheus will eat all available memory if you let it. MongoDB will cache everything in RAM. Kafka needs fast disks for log segments. Plan for 2x the resources you think you need, especially for storage.

The real secret: this stack works great once you get through the initial configuration hell and learn from your production failures.

Forget those perfect YAML configs you see in Medium articles written by people who've never seen production traffic. Here's what it actually takes to get Kafka, MongoDB, Kubernetes, and Prometheus working together without everything catching fire, including all the stuff that breaks when you least expect it.

Kubernetes Setup That Won't Fall Over

Resource Reality Check: The official docs claim you need 3 nodes with 8GB RAM each. That's complete bullshit designed to make their toy demos work. After watching our cluster die repeatedly with those specs, here's what you actually need if you don't want to get woken up every night:

• Worker nodes: 16GB RAM minimum, 8 vCPUs each (learned this after nodes kept crashing)
• Storage: 500GB fast SSDs per node (NVMe if you can afford it - trust me, slow disks will ruin your life)
• Network: 10Gbps networking or Kafka will bottleneck and make everything crawl

Install the basics, but expect at least half of this shit to fail on the first try:

## Prometheus Operator (this will fail the first time)
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/main/bundle.yaml

## KEDA for autoscaling (works better than HPA for event-driven stuff)
kubectl apply -f https://github.com/kedacore/keda/releases/download/v2.12.0/keda-2.12.0.yaml

Gotcha #1: The Prometheus Operator will fail spectacularly on EKS because of RBAC bullshit that nobody mentions in the docs. You'll spend 3 hours creating service accounts and role bindings while wondering why you didn't just use DataDog.

Kafka: Where Dreams Go to Die

Be Careful With Newer Kafka Versions: I learned this lesson after newer versions caused production issues. Some versions have had consumer group stability problems that can crash your brokers under load after a few days. Stick with versions that have been stable for a few months until newer ones are proven in production.

Here's a Strimzi config that actually works:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: production-kafka
spec:
  kafka:
    version: 3.5.1  # Use stable versions
    replicas: 3
    config:
      # These settings matter for performance
      num.network.threads: 8
      num.io.threads: 16
      socket.send.buffer.bytes: 102400
      socket.receive.buffer.bytes: 102400
      socket.request.max.bytes: 104857600
      # Replication settings that won't lose data
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      min.insync.replicas: 2
      # Log retention that won't fill your disks
      log.retention.hours: 168
      log.retention.bytes: 1073741824
    storage:
      type: persistent-claim
      size: 500Gi
      storageClass: gp3  # Use gp3, not gp2
    # JMX for Prometheus - this is critical
    jmxOptions:
      javaOptions:
        - \"-javaagent:/opt/jmx_exporter/jmx_prometheus_javaagent.jar=9404:/opt/jmx_exporter/kafka-config.yml\"
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 100Gi
      storageClass: gp3

Gotcha #2: Strimzi's default JMX config is absolute trash. The JMX exporter will vomit 500+ metrics and murder your Prometheus server with high cardinality bullshit. Use a custom JMX config or prepare for your monitoring to die a slow, painful death.

MongoDB: Connection Pool Hell

MongoDB looks deceptively simple until you try to actually monitor the bastard. The official MongoDB Kubernetes operator works okay, but extracting useful metrics is where you'll lose your sanity.

apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
metadata:
  name: mongodb-replica-set
spec:
  members: 3
  type: ReplicaSet
  version: \"7.0.4\"
  security:
    authentication:
      modes: [\"SCRAM\"]
  users:
    - name: mongodb-admin
      db: admin
      passwordSecretRef:
        name: mongodb-admin-password
      roles:
        - name: clusterAdmin
          db: admin
        - name: userAdminAnyDatabase
          db: admin
        - name: dbAdminAnyDatabase
          db: admin
  additionalMongodConfig:
    storage.wiredTiger.engineConfig.journalCompressor: snappy
    storage.wiredTiger.collectionConfig.blockCompressor: snappy
    net.maxIncomingConnections: 65536

Gotcha #3: Some versions of the Percona MongoDB exporter have had issues with auth enabled. If you're having problems with MongoDB metrics disappearing, try an older stable version of the exporter - I've had better luck with earlier releases.

Gotcha #4: MongoDB's default connection pool is way too small for anything beyond a basic app. Set maxPoolSize to something like 200 and minPoolSize to 50, or watch your services timeout during any real load.

Prometheus: The Memory Monster

Prometheus Will Eat Your RAM: Prometheus keeps all active time series in memory for fast querying, which means RAM usage explodes faster than your credit card debt. Plan for 16-64GB if you don't want it to OOM every other day.

Prometheus is a RAM-hungry monster that will consume everything if you don't keep it on a leash. Here's how to not let it eat your entire server:

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
      evaluation_interval: 15s
    
    # Scrape config that doesn't kill Prometheus
    scrape_configs:
      # Kafka JMX metrics (high frequency, low cardinality)
      - job_name: 'kafka-brokers'
        kubernetes_sd_configs:
        - role: pod
        relabel_configs:
        - source_labels: [__meta_kubernetes_pod_label_strimzi_io_name]
          action: keep
          regex: production-kafka-kafka
        - source_labels: [__address__]
          action: replace
          regex: ([^:]+):.*
          replacement: ${1}:9404
          target_label: __address__
        scrape_interval: 10s  # Kafka metrics are fast-changing
        
      # MongoDB metrics (lower frequency)
      - job_name: 'mongodb-exporter'
        static_configs:
        - targets: ['mongodb-exporter:9216']
        scrape_interval: 30s  # MongoDB metrics change slowly

Resource Planning Reality (aka what I wish someone told me before Prometheus ate our entire server):

• Prometheus: 32GB RAM minimum, 64GB for serious workloads - I tried running it with 16GB and it OOMKilled every 6 hours
• Storage: Plan 1-2GB per day per 1000 active series - we hit 10GB/day with just basic metrics, storage costs killed our budget
• CPU: 8 cores minimum because Prometheus is single-threaded for queries and will peg one core at 100% during complex queries

Gotcha #5: Prometheus recording rules are absolutely fucking mandatory. Pre-aggregate your high-frequency metrics or watch your Grafana dashboards timeout while you're demo'ing to the CEO:

rules:
  - name: kafka.aggregation
    rules:
    - record: kafka:broker_messages_per_sec:rate5m
      expr: sum(rate(kafka_server_brokertopicmetrics_messagesin_total[5m])) by (instance)
    - record: kafka:consumer_lag_total:sum
      expr: sum(kafka_consumer_lag_sum) by (consumergroup, topic)

The Integration Nightmare

Getting all this shit to talk to each other is where the real nightmare begins:

Service Discovery: Kubernetes DNS is flaky as hell. Use headless services for database connections or enjoy random connection failures:

apiVersion: v1
kind: Service
metadata:
  name: mongodb-headless
spec:
  clusterIP: None
  selector:
    app: mongodb
  ports:
  - port: 27017

Network Policies: If you have NetworkPolicies enabled (you should), remember to allow traffic between:

• Kafka brokers (port 9092, 9093)
• Prometheus to all exporters (various ports)
• Applications to MongoDB (27017)
• JMX exporters (9404, 9216, etc.)

Persistent Volumes: Use separate storage classes for different workloads:

• Kafka: Fast SSDs (gp3 with high IOPS)
• MongoDB: Balanced SSDs (gp3 standard)
• Prometheus: Fast SSDs for WAL, slower for blocks

What Actually Breaks in Production

Kafka Consumer Lag Spikes: 90% of the time this is your JVM doing stop-the-world garbage collection. Monitor GC metrics and fix your heap settings. Use G1GC - CMS is ancient garbage that should have died years ago.

MongoDB Replica Set Bullshit: Primary elections always happen at the worst possible time, usually during AWS maintenance windows or Black Friday. Your connection strings better include the full replica set or you'll lose writes and get blamed for the outage. Configure read preferences and write concerns correctly or suffer.

Prometheus OOMKilled Again: High cardinality metrics are Prometheus kryptonite. Some genius will add user IDs as labels and murder your monitoring. Avoid any unbounded labels or prepare for 3am pages.

Kubernetes Pod Evictions: If you don't set resource requests and limits properly, Kubernetes will randomly evict your pods during memory pressure and ruin your weekend. Set requests to 80% of expected usage, limits to 150%, or get comfortable with unstable services.

Cost Reality (2025 Pricing)

This stack isn't cheap, and AWS will drain your bank account faster than you can say "observability". Here's what it actually costs after I got our first bill and nearly choked:

• EKS Cluster: $73/month base (that's just for the privilege of using K8s)
• 3x m5.2xlarge workers: $280/month each = $840/month (you need this much RAM or everything dies)
• EBS gp3 storage: $0.08/GB/month (3TB total = $240/month, grows fast with retention)
• Data transfer: $20-50/month (until your traffic spikes and AWS charges you $200)
• Load balancers: $16/month each (you'll need 3-4 before you realize it)

Total monthly cost: $1200-1500/month for a "basic" production setup that will still break

Scale this up for enterprise workloads and you're looking at $5000-10000/month easily. Your CFO will ask what the fuck you're spending money on.

The Nuclear Options (When Everything Breaks)

Kafka Cluster Restart: Sometimes you need to restart the entire cluster:

kubectl scale statefulset production-kafka-kafka --replicas=0
kubectl scale statefulset production-kafka-kafka --replicas=3

MongoDB Replica Set Reset: When replica set gets corrupted:

// Connect to any working member
rs.reconfig(config, {force: true})

Prometheus Storage Cleanup: When Prometheus runs out of disk:

kubectl exec prometheus-0 -- rm -rf /prometheus/data/*
## Restart Prometheus pod
kubectl delete pod prometheus-0

Complete Restart Order: If everything is fucked:

1. Stop applications
2. Scale down Kafka Connect
3. Scale down Kafka brokers
4. Scale down MongoDB
5. Restart in reverse order
6. Pray to whatever deity you believe in

The key to success: monitor everything, alert on what matters, and have runbooks for when (not if) things break.