Apache Pulsar - Multi-Layered Messaging Platform

Pulsar is what happens when you decide Kafka's architecture is fundamentally broken and rebuild everything from scratch. Yahoo created it around 2013 because they needed something that could scale beyond Kafka's limitations. The current version is 4.1.0 released September 8, 2025, and it actually works - if you can handle the operational complexity.

The Architecture That Makes Your Life Complicated

Pulsar's layered architecture separates compute from storage, which is brilliant in theory. In practice, you're now running two distributed systems instead of one.

You've got four moving parts to worry about:

• Pulsar Brokers: These handle routing but don't store data. When they crash, your topics just migrate to other brokers. Sounds nice until you realize that the migration can take 30+ seconds and your clients start timing out.
• Apache BookKeeper: This is where your data actually lives. When BookKeeper has issues, you're in for a long night. And trust me, you'll see errors like BKNotEnoughBookiesException at 3am and wonder why you didn't just use Kafka.
• Apache ZooKeeper: Because every distributed system needs ZooKeeper to make your life miserable. It'll work fine until you hit ~500K topics, then ZK becomes your bottleneck and you'll be optimizing JVM heap sizes.
• Pulsar Functions: Serverless stream processing that works great in demos, terrible for debugging in production. When a function fails, good luck figuring out which K8s pod it was running in.

The Multi-Tenancy Promise

Pulsar's multi-tenancy is actually pretty good. You get tenant isolation without running separate clusters, which saves on ops overhead. But good luck debugging cross-tenant issues when they happen.

Scale Reality Check

Can Pulsar handle millions of topics? Probably. Should you do that? Probably not unless you have a dedicated platform team. The theoretical limits are impressive, but the operational reality is that most people run into ZooKeeper bottlenecks long before they hit Pulsar's actual limits.

Yahoo runs this thing at massive scale, and it works for them. Whether it'll work for your use case depends entirely on whether you're willing to invest in understanding BookKeeper and ZooKeeper operational patterns.

Geo-Replication: Great Until It Breaks

Pulsar's geo-replication works, but it's not magic. You're replicating entire message streams across regions, which means your network bills are going to hurt. When replication falls behind, you'll spend hours trying to figure out if it's a network issue, BookKeeper problem, or just the eventual consistency model working as designed.

I once had geo-replication silently fail for 6 hours because of a misconfigured security group rule. The cluster logs showed everything was "healthy", but messages just weren't making it across regions. Took down our disaster recovery capabilities without a single alert firing.

Load Balancing That Actually Works

Unlike Kafka's manual partition rebalancing nightmare, Pulsar's load balancing is genuinely good. Brokers automatically split topic bundles when they get hot, and adding new nodes actually distributes load without requiring manual intervention. This is one of the few areas where Pulsar delivers on its promises.

Tiered Storage: The Feature You'll Wish You'd Set Up Earlier

Tiered storage to S3/GCS is brilliant for cost control. Old messages automatically move to cheap storage while recent data stays fast. Setting it up requires understanding BookKeeper's ledger lifecycle, but once it's working, it saves serious money on long-term retention.

Word of warning: make sure you get the S3 bucket policies right the first time. I've seen production clusters lock up trying to offload data to S3 with misconfigured IAM roles, throwing OffloadException: Access Denied errors that took hours to debug.

Client Libraries: Java Works, Others Are Hit-or-Miss

The official clients are:

• Java: Rock solid, all features work
• Go: Pretty good, occasionally missing newer features
• Python: Adequate for most use cases, asyncio support exists
• C++: Fast but documentation is sparse
• Node.js: Works but you'll hit edge cases. The connection pooling breaks in weird ways under high load.
• C#: Basic functionality only. Don't expect schema registry support anytime soon.

If you're not using Java, test thoroughly before committing to production. I've had Python clients randomly start consuming duplicate messages in production due to some obscure consumer acknowledgment bug that only shows up under load.

Pulsar Functions: Cool Demo, Production Hell

Pulsar Functions let you deploy stream processing code directly into the Pulsar cluster. It's serverless in theory, but debugging a function that's misbehaving in production involves digging through Kubernetes logs and understanding Pulsar's function runtime. Most people end up running separate stream processing anyway.

Schema Registry: Built-In But Basic

The schema registry supports Avro, JSON, and Protobuf. Schema evolution works for simple cases, but complex schema migrations require careful planning. It's not as sophisticated as Confluent's Schema Registry, but it's included and mostly works.

Security: Comprehensive If You Can Configure It

Pulsar's security model is actually quite good:

• TLS everywhere (required for production)
• JWT/OAuth integration that works
• Fine-grained ACLs down to the topic level
• End-to-end encryption for sensitive data

The challenge is getting all the moving parts configured correctly. BookKeeper, ZooKeeper, and Pulsar brokers all need coordinated security config, and getting it wrong means either no security or complete lockout.

The Companies Actually Using It

I've talked to engineers at several companies running Pulsar in production. Here's what they actually told me:

Yahoo: The Success Story That Started It All

Yahoo built Pulsar because Kafka couldn't handle their scale. They've been running it since 2015, and it works for them because:

• They had the team that built it
• They invested heavily in operational expertise
• Their use case (massive multi-tenant messaging) is exactly what Pulsar was designed for

Tencent: Scale But With Pain Points

Tencent runs Pulsar at massive scale, but their optimization blog posts tell the real story:

• They had to optimize client connection pooling because the default configuration doesn't scale
• ZooKeeper became a bottleneck at around 600K topics
• They spent significant engineering effort on stability improvements

They handle "tens of billions of transactions during peak time" according to verified metrics, which is impressive but not the inflated numbers you sometimes see in marketing materials.

Flipkart: Topic-as-a-Service Reality

Flipkart built their platform on Pulsar's multi-tenancy, but it took them 18 months to get it production-ready. Their main challenges:

• Getting geo-replication working reliably
• Building monitoring and alerting for BookKeeper
• Training their ops team on three different distributed systems

The IoT Use Case: Where Pulsar Actually Shines

Cisco's IoT platform is one of the success stories that makes sense. IoT workloads need:

• Massive numbers of topics (one per device)
• Multi-tenancy (different customers)
• Geo-distribution

This is exactly what Pulsar was designed for. But notice they replaced "legacy message queue services" - they didn't migrate from Kafka.

What Companies Don't Tell You

The Migration Stories That Aren't
Most Pulsar success stories involve greenfield deployments or replacing older systems. Kafka-to-Pulsar migrations are rare because:

• The operational overhead is significant
• Kafka works well enough for most use cases
• The business case for migration is hard to justify

The Platform Team Requirement
Every successful Pulsar deployment I've seen has a dedicated platform team. If you don't have 2-3 engineers who can become Pulsar experts, you're going to struggle.

Common Use Case Patterns

Multi-Tenant SaaS Platforms
This is where Pulsar really wins. If you're building a platform that serves multiple customers and need topic isolation, Pulsar's multi-tenancy is genuinely better than Kafka's manual approaches.

IoT Data Ingestion
Millions of topics? Pulsar handles this better than alternatives. Each device gets its own topic, and the routing/load balancing just works.

Event-Driven Microservices
Pulsar works fine for this, but so does Kafka. Choose based on operational capacity, not technical features.

The Reality Check

Pulsar works well for the companies using it, but they all have something in common: significant investment in platform engineering. If you're evaluating messaging systems, ask yourself:

• Do you actually need multi-tenancy? (Hint: you probably don't)
• Can you invest in the operational expertise? (Budget for 2-3 senior engineers who'll become Pulsar experts)
• Are you building something that will justify the complexity? (Most aren't)

For most companies, Kafka is boring and reliable. Pulsar is powerful but complex. Choose accordingly. And if your boss is pushing Pulsar because they read a blog post about Yahoo's scale, remind them that Yahoo also had the team that built the fucking thing.