Kafka + Spark + Elasticsearch: Don't Let This Pipeline Ruin Your Life

Still here after that warning? Good. Look, this stack became popular because it actually works at scale. Lambda functions shit the bed after 100K events, and simple queues are great until you need to transform the data or search through it later.

The Three-Headed Monster You're About to Build

You've got three moving parts that all hate each other: Kafka (the data firehose), Spark (the processing engine that eats RAM), and Elasticsearch (the search engine that corrupts indices for fun).

This architecture is awesome if you have Netflix's budget. For everyone else, expect your AWS bills to get ugly fast - like, maybe $8k monthly, could be way higher depending on your data volume. Costs have been climbing since 2023 with all the instance price increases.

I learned this the hard way when our "simple" event tracking system went from around $2k to like $12k+ monthly - maybe higher, I stopped looking at the exact number after it hit five figures. Turns out nobody mentioned that Elasticsearch replicates everything 3 times by default. Fun discovery.

This shit gets expensive. Your typical production setup needs beefy instances across all three services, plus storage, plus network bandwidth. Start budgeting at least $8k monthly, probably more once you factor in monitoring, backups, and the extra capacity you'll need when things go sideways.

What Each Component Actually Does

Kafka: The Data Firehose
Kafka ingests data and keeps it around for a while. The partition-based architecture sounds great until you realize wrong partitioning will bottleneck your entire pipeline. Pro tip: start with more partitions than you think you need - you can't increase them easily later without major pain.

The Kafka cluster architecture distributes data across multiple brokers for fault tolerance. Each topic partition gets replicated across configured brokers, but replication complexity can bite you in production.

Version hell warning: Kafka 3.6.1 works great, but then 3.6.2 came out and broke a bunch of stuff - consumer group rebalancing got weird and offset management had issues. Kafka 4.x series (current as of Sep 2025) has breaking changes in the admin client API and consumer group management. Don't ask why, nobody knows. Pin every single version in your build files or you'll spend days debugging ClassNotFoundException errors when Gradle decides to "helpfully" upgrade dependencies.

The new KRaft mode (no more ZooKeeper dependency) in Kafka 4.x is supposedly stable but requires different monitoring and operational procedures. Check the Kafka compatibility matrix religiously before upgrading anything.

Spark: The RAM Gobbler
Spark Structured Streaming processes your data in "micro-batches" which is fancy talk for "we'll batch it up and pretend it's real-time." The docs claim sub-second latency. In reality, expect 2-5 seconds once you add actual business logic.

The micro-batch processing model achieves 100ms latency at best, but continuous processing can hit 1ms if you're willing to sacrifice fault tolerance. Databricks recommends micro-batching for most use cases due to exactly-once guarantees.

Spark's memory management will drive you insane. Set spark.executor.memory=8g and it'll use 12GB. The checkpointing mechanism that "enables recovery from failures" fails randomly and corrupts your state. Always backup to multiple locations. Read Spark performance tuning guide before you lose your sanity.

Elasticsearch: The Index Destroyer
Elasticsearch will eat all your RAM and ask for more. The distributed architecture supports "petabyte-scale search" if you have petabyte-scale hardware budgets.

The cluster, nodes, and shards architecture distributes data across primary and replica shards, but shard allocation can become a nightmare at scale. Too many shards destroy performance, too few create hotspots.

Real talk: Elasticsearch decides to go read-only when disk hits 85% full. This happens on weekends when you're not watching. Your pipeline stops, data backs up in Kafka, and you get angry calls from product managers. Monitor cluster health religiously or suffer the consequences.

The Data Flow (When It Works)

1. Applications → Push events to Kafka topics (works great until network blips)
2. Kafka Brokers → Store and replicate data (replication factor saves you from disk failures)
3. Spark Jobs → Read from Kafka using Kafka-Spark connector (breaks every upgrade)
4. Data Processing → Transform and enrich (where 90% of your bugs live)
5. Elasticsearch → Index for search (randomly stops working and nobody knows why)
6. Kibana → Pretty dashboards (when Elasticsearch cooperates)

What the Docs Don't Tell You

The marketing bullshit about "100x faster than batch" is assuming your data is pristine, your network never hiccups, and AWS gives you unlimited resources for free. Real world? Your data is garbage, your network drops packets, and your AWS bill makes CFOs cry.

Real performance depends on:

• How much crap is in your data (answer: a lot)
• Network latency between your components (AWS AZs add 1-3ms)
• Whether Elasticsearch feels like working today (coin flip - check indexing performance issues)
• Phase of the moon and your proximity to a data center

Plan to hire someone whose job is just keeping this shit running. Kafka rebalances for no reason, Spark eats all your RAM, ES corrupts indices on Tuesdays. Budget for at least one full-time babysitter who actually knows what they're doing.

Look into production monitoring strategies before you deploy this monster. The stream processing challenges are real and the learning curve will kick your ass.

The Implementation That'll Only Break Half the Time

Building this pipeline will take 6 months if you're lucky, 2 years if you're not. Here's what actually works in production, not the demo bullshit from conferences.

Dependencies That'll Break Your Build

Maven Dependencies (that actually work together)

<dependencies>
    <!-- Use these versions - I wasted 3 days testing combinations so you don't have to -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql-kafka-0-10_2.12</artifactId>
        <version>3.5.2</version> <!-- 3.5.1 had weird streaming bugs -->
    </dependency>
    <dependency>
        <groupId>org.elasticsearch</groupId>
        <artifactId>elasticsearch-spark-30_2.12</artifactId>
        <version>8.15.2</version> <!-- Newer versions have issues, stick with this -->
    </dependency>
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka-clients</artifactId>
        <version>3.8.0</version> <!-- Latest that doesn't break -->
    </dependency>
</dependencies>

Use these versions. I spent 3 days figuring out what actually works together so you don't have to. Spark 3.5.1 has some weird streaming bugs that the error messages don't explain properly, and ES 8.12 through 8.14 had memory leaks in the bulk indexing API that would kill your cluster overnight. Found that out the hard way.

Kafka Setup (The Part That'll Bite You)

Producer Configuration that doesn't lose data:

## Connection settings (obvious stuff)
bootstrap.servers=kafka-broker-1:9092,kafka-broker-2:9092

## This is where you make trade-offs
acks=all  # Use \"all\" not \"1\" unless you enjoy data loss
batch.size=65536  # Higher = better throughput, worse latency
linger.ms=5  # Wait 5ms to batch messages (adds latency but saves your brokers)
compression.type=snappy  # LZ4 is faster but Snappy compresses better
max.in.flight.requests.per.connection=5
enable.idempotence=true  # Prevents duplicate messages (mostly)

Topic Creation (that won't fuck your scaling):

## Start with more partitions than you think you need
kafka-topics.sh --create \
  --topic streaming-events \
  --bootstrap-server localhost:9092 \
  --partitions 24 \  # You can't easily increase this later
  --replication-factor 3 \  # 2 is gambling, 3 is safe
  --config retention.ms=604800000  # 7 days, adjust based on your storage budget

Pro tip from 3am debugging sessions: Start with 2x as many partitions as you think you need. You can't increase partitions without re-keying all your data, which is a multi-day clusterfuck. Read about Kafka topic partitioning strategies and scaling considerations before you create topics.

Spark Implementation (The Memory-Eating Monster)

Core Streaming Application (that will OOM your cluster):

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._

// This looks clean but will OOM your cluster if configured wrong
val spark = SparkSession.builder()
  .appName(\"KafkaSparkElasticsearch\")
  .config(\"spark.sql.adaptive.enabled\", \"true\") // Pray this actually helps
  .config(\"spark.sql.adaptive.coalescePartitions.enabled\", \"true\")
  .config(\"spark.executor.memory\", \"8g\") // It'll use 12g anyway
  .config(\"spark.executor.memoryFraction\", \"0.8\") // Controls how much goes to caching vs execution
  .config(\"spark.serializer\", \"org.apache.spark.serializer.KryoSerializer\") // Faster than default
  .getOrCreate()

// Define your schema or Spark will guess wrong and fuck everything up
val schema = StructType(Array(
  StructField(\"timestamp\", TimestampType, true),
  StructField(\"user_id\", StringType, true),
  StructField(\"event_type\", StringType, true),
  StructField(\"properties\", MapType(StringType, StringType), true)
))

val kafkaStream = spark.readStream
  .format(\"kafka\")
  .option(\"kafka.bootstrap.servers\", \"kafka-broker-1:9092,kafka-broker-2:9092\")
  .option(\"subscribe\", \"streaming-events\")
  .option(\"startingOffsets\", \"latest\") // Use \"earliest\" if you want to reprocess everything
  .option(\"failOnDataLoss\", \"false\") // Set to true in prod once your pipeline is stable
  .option(\"maxOffsetsPerTrigger\", \"10000\") // Limit batch size or you'll OOM
  .load()

// This is where your pipeline will break 90% of the time
val processedStream = kafkaStream
  .select(from_json(col(\"value\").cast(\"string\"), schema).as(\"data\"))
  .select(\"data.*\")
  .filter(col(\"user_id\").isNotNull) // Filter out garbage early
  .withColumn(\"processed_time\", current_timestamp())
  .withWatermark(\"timestamp\", \"10 minutes\") // Allow 10 min late data, drop the rest
  .groupBy(window(col(\"timestamp\"), \"1 minute\"), col(\"event_type\"))
  .agg(count(\"*\").as(\"event_count\"))

Real talk: The maxOffsetsPerTrigger setting will save your ass. Without it, Spark will try to process 10 million messages in one batch and kill your cluster. Start with 1000, increase gradually while watching memory usage. Check Spark streaming performance tuning for more memory optimization tricks.

The failOnDataLoss option is set to false because Kafka retention will bite you. Set it to true only after you've tuned retention policies and know your backpressure patterns. The Databricks streaming guide explains exactly-once semantics and when they actually work (spoiler: not always).

Elasticsearch Integration (The Index Corruptor)

Elasticsearch Sink (that randomly stops working):

// These settings took 6 months of trial and error to get right
val esOptions = Map(
  \"es.nodes\" -> \"elasticsearch-node-1,elasticsearch-node-2,elasticsearch-node-3\",