Burrow - Kafka Consumer Lag Monitoring That Actually Works

Consumer lag monitoring is a nightmare. I've wasted weeks tuning thresholds that either fire constantly during traffic spikes or miss dead consumers for hours.

First time I set up Kafka monitoring, I used JMX metrics and Prometheus. Set the lag threshold to like 10,000 messages because that's what some tutorial said. Black Friday hits, traffic doubles, every single alert fires at 2AM. Ops team is fucking pissed. So I raise the threshold to 50,000 or something. Few weeks later, a consumer dies on Thursday afternoon and sits dead all weekend because something like 45,000 messages of lag was "fine" according to my brilliant threshold.

So Burrow does this clever shit - it reads the __consumer_offsets topic that Kafka uses internally to track where consumers are. No per-app JMX bullshit setup, no config for each consumer group, no "we forgot to monitor the payments service" disasters. Point Burrow at your cluster and it sees everything automatically.

The sliding window thing is what actually makes it not suck. Instead of "lag > threshold = bad," it looks at the pattern over time. High lag but dropping? Consumer's probably catching up from a restart or batch job. High lag and climbing? Falling behind. High lag and flatlined while new messages pile up? Consumer's fucking dead, go fix it.

The Consumer Monitoring Shitshow I Lived Through

Threshold Tuning Hell

I've spent three different weeks at three different jobs tuning consumer lag thresholds. Every time, same story. Start with 1,000 messages like every tutorial recommends - alerts fire on every deploy when consumers restart and rebalance. Raise to 10,000 based on capacity planning docs - Black Friday traffic spike wakes the whole team up. Raise to 100,000 out of desperation - miss actually dead consumers for days.

There's no sweet spot. Traffic patterns aren't consistent, batch processing creates natural spikes, and consumers have different processing speeds depending on message complexity. One threshold can't cover all scenarios.

JMX Disappearing Act

JMX consumer metrics vanish the moment your consumer process dies. Which is exactly when you need them most. Why the fuck does the monitoring disappear right when something breaks? I've spent more time debugging "why aren't we getting lag metrics" than fixing actual broken consumers. The monitoring system can't tell you about problems it can't see.

The "Forgot to Monitor" Problem

New service deploys, team "forgets" to add JMX endpoint configuration. Happened to us last year - some user activity processor died for like three weeks and nobody noticed until the monthly report was empty. "Why didn't we know this stopped?" Because nobody remembered to set up monitoring for the new service. Happens every fucking time in large organizations.

Numbers Without Context Are Useless

Consumer lag of like 50,000 messages. Should you panic? Maybe the consumer's processing a daily batch job and working through backlog normally. Or maybe it's been dead for the last 2 hours. The number by itself tells you jack shit.

How Burrow Fixes This Mess

The key insight: stop asking consumers how they're doing (they lie or disappear). Read Kafka's internal bookkeeping instead.

Every time a consumer commits an offset, Kafka writes it to the __consumer_offsets topic. Burrow reads this topic directly, so it sees every consumer group whether they're running, dead, or somewhere in between. No JMX setup, no per-app configuration, no gaps.

The Sliding Window Thing

Here's what makes it not suck. Instead of "lag > 50,000 = alert," Burrow looks at the last 10 offset commits over about 10 minutes. If lag is high but the commits are moving forward and getting closer to the current offset, the consumer is catching up. Maybe it restarted, maybe it's working through a batch job. Either way, it's not dead.

But if the lag is high and the commits stopped advancing while new messages keep arriving? Consumer's dead. The sliding window catches this pattern reliably.

Three States I Actually Understand

• OK: Consumer is keeping up or catching up normally
• WARNING: Falling behind but still making progress
• ERROR: Consumer is stalled, dead, or completely broken

I query the HTTP API from my existing Prometheus setup. Same alerting infrastructure, just better consumer status evaluation. Works with any Kafka since 0.8.2 when __consumer_offsets became a thing.

This is the monitoring setup that finally let me sleep through the night without lag alerts going off every time traffic spiked or a batch job started.

The Sliding Window Evaluation (Actually Clever)

The sliding window thing is what makes Burrow actually useful compared to threshold hell. Here's what it watches:

Commit Progression: Is the consumer regularly committing offsets that are moving forward? If offsets are stuck but messages keep arriving, the consumer is probably dead or stuck processing one bad message. Poison pill messages can kill consumers for hours while offsets don't advance.

Lag Trend Over Time: This is the key insight. High lag + decreasing = consumer is catching up from a batch job or restart. High lag + increasing = consumer is falling behind processing speed. High lag + flatlined = consumer is dead and you need to go fix it.

Pattern Recognition: During normal traffic spikes, healthy consumers show consistent sawtooth patterns. Random erratic lag jumps usually mean something's fucked - JVM GC pauses (especially with Java 8's G1GC before tuning), network partition issues, thread pool exhaustion in Spring Boot 2.x apps, or code bugs that cause intermittent failures.

The default window is 10 offset commits over ~10 minutes, which works well for most setups. You can tune it, but honestly the defaults are pretty solid.

Multi-Cluster Support (Because You Have Dev/Stage/Prod)

One Burrow instance can monitor multiple Kafka clusters, which is actually useful instead of just marketing bullshit:

Separate Cluster Monitoring: Each cluster gets its own consumer tracking and evaluation. Your dev cluster's fucked-up consumers won't affect production monitoring. Burrow keeps the data separate so cross-cluster noise isn't an issue.

Same API for Everything: /v3/kafka/{cluster} endpoints work the same across all clusters. Your Prometheus monitoring scripts or Grafana dashboards don't need to know which cluster they're hitting - just change the cluster name in the URL.

Scales Pretty Well: LinkedIn runs this on thousands of consumer groups across multiple data centers. A single Burrow instance handles way more than you probably need, unless you're Netflix-scale with petabytes of data or Uber-level real-time processing. Our instance monitors like 200+ consumer groups across dev/stage/prod with zero performance issues since we set it up.

HTTP API (Simple and Actually Works)

The API is straightforward - no GraphQL bullshit, no complex authentication schemes, just HTTP GET requests:

GET /v3/kafka                                    # List clusters
GET /v3/kafka/{cluster}/consumer                 # List consumer groups
GET /v3/kafka/{cluster}/consumer/{group}/status  # The money shot - consumer status
GET /v3/kafka/{cluster}/topic                   # List topics

The /status endpoint returns JSON with the important bits: overall status (OK/WARNING/ERROR), per-partition details, and the lag trends that drove the decision. Perfect for integrating with Prometheus, Grafana, or whatever monitoring stack you already have.

Alerting (Email and HTTP Webhooks)

Burrow can push alerts instead of just providing an API:

Email: Basic SMTP alerts. Honestly, most people skip this and just poll the API from their existing monitoring system instead of dealing with email routing.

HTTP Notifications: POSTs JSON to whatever endpoint you want. Useful for sending alerts to PagerDuty, Slack webhooks, or your custom alerting system. Includes configurable templates and retry logic.

Most deployments just use the HTTP API and skip the built-in notifications - easier to integrate with whatever alerting system you already have than to configure another notification pathway.

Deployment Reality Check

Resources: It's Go, so it's not a memory hog like JVM apps. Typical production instance uses like 500MB RAM and minimal CPU. Scales pretty well with number of consumer groups you're monitoring.

High Availability: No built-in clustering, so you run multiple instances behind a load balancer. Since it's stateless (just reads from Kafka), this works fine. Most people run 2-3 instances across different AZs.

Configuration: TOML files for config. You'll need to restart when changing config, but the state is just in-memory sliding windows, so restart is fast.

Docker: There's an official Docker image and docker-compose setup that actually works. The compose includes Kafka and Zookeeper for testing, which is handy.

Real Gotcha: Burrow needs read access to __consumer_offsets. If your Kafka cluster has ACLs, make sure Burrow can read that topic, or you'll get the cryptic error TOPIC_AUTHORIZATION_FAILED and mysteriously see "no consumer groups found" in the API. I spent like 4 hours debugging this once before realizing the security team had locked down internal topics. That was a fun afternoon.

The __consumer_offsets Deep Dive: This internal topic is where Kafka stores consumer group metadata and offset commits. By default it has 50 partitions (configurable with offsets.topic.num.partitions) and uses compacted cleanup policy to keep the latest offset for each consumer group + partition combination. Burrow essentially becomes another consumer of this topic, which is why it can see all consumer activity without any per-app configuration.