Why Apache Pulsar? Part 1

Apache Pulsar (incubating) is a next-generation pub/sub messaging system developed at Yahoo. Pulsar was developed from the ground up to address several shortcomings of existing open source messaging systems and has been running in production for three years, powering critical applications like Yahoo! Mail, Yahoo! Finance, Yahoo! Sports, Flickr, the Gemini Ads Platform, and Sherpa, Yahoo’s distributed key value store. Pulsar was open sourced in late 2016 and is currently undergoing incubation under the auspices of the Apache Software Foundation. In this blog post, we will highlight key features of Pulsar that are available out of the box.

In a previous blog post, we introduced several pulsar concepts and terminology. A Pulsar cluster is composed of a set of brokers and bookies and an Apache ZooKeeper ensemble for coordination and configuration management. The Pulsar broker is the component that receives, stores and delivers messages. The bookies are from Apache BookKeeper that provide durable storage for messages until they are consumed. A high level architectural diagram is shown in Figure 1

^{Figure 1. High-level view of a Pulsar cluster}

Flexible Messaging Model

There are two traditional messaging models: queuing and publish-subscribe. Queuing is a point-to-point communication model - a pool of consumers may read from a server and each message is delivered to one of them - it allows you to divide up the processing of data over multiple consumer instances and scale your processing. Publish-subscribe is a broadcast communication model - a message is broadcast to all the consumers.

Pulsar generalizes these two messaging concepts through one unified messaging API - Producers publish messages to topics, messages are broadcast to different subscriptions. Consumers can then subscribe to those subscriptions to consume messages. The consumers in same subscription can choose a flexible way to consume messages - exclusively, shared and failover. As with a queue, shared subscription, with round-robin delivery, allows applications to divide up processing over the consumers in same subscription. Unlike other messaging systems, Pulsar allows scaling the number of active consumers beyond the number of partitions within a topic.

Since Pulsar is using Apache BookKeeper as the stream storage, it also exposes the underlying log through a stream Reader API, allowing applications to start consuming from any earlier positions.

Flexible Deployment Modes

Pulsar is flexible to deploy in different environment. Pulsar can be run on bare-metal machines, Kubernetes cluster on prem and in the cloud, Google Kubernetes Engine and AWS. One can download and use Pulsar in a single node as well for development and testing purposes. In this case, the entire Pulsar broker, bookie and ZooKeeper all run in a single process.

Multi-Tenancy

Right from the start, Pulsar was designed for deployment as a hosted service for private and public cloud, which was not offered by any available open source system. For medium to large organizations, it is much more cost effective to share a single deployment of Pulsar rather than requiring each team to operate their own messaging solution. While each team may find it easy enough to set up and deploy their own messaging solution, it’s usually far more difficult to operate those solutions, as they often require in-depth knowledge of system internals to configure, monitor, and troubleshoot them effectively. Furthermore, by sharing a single deployment across the organization, it is cost effective since the cluster servers are better utilized, sharing of a dev-ops team by multiple teams and effective capacity planning using expected peak usage combined with projected growth.

Geo-Replication

Unlike other messaging systems, Pulsar offers geo-replication as a first-class feature. Users can easily enable replication of message data between Pulsar clusters by configuring the geographical regions where to which topics need to be replicated and Pulsar automatically takes care of the rest. Data is continuously replicated to the remote Pulsar clusters. In the case of network failures across data centers, the data is stored and retried until the replication is successful.

Pulsar can replicate data simultaneously to other Pulsar clusters in multiple data centers located in different geographical regions. At Yahoo, the key-value store Sherpa uses Pulsar for replicating its Write-ahead Logs (WALs). Logs are replicated up to 10 geographical regions to achieve eventual consistency in the key-value store. Geo-replication can be deployed flexibly in several configurations. You can replicate across private clouds of an organization, between private and public clouds, or between public clouds and even from edge data centers to private or public clouds. Large organizations that are looking to move to the cloud can deploy Pulsar across different cloud providers and replicate data across them without tying themselves to proprietary cloud provider APIs.

Durability

Once Pulsar receives data and acknowledges (“acks”) it, it guarantees that the data is never lost in the event of errors, power failures, etc. Durability guarantees are determined by the number of disks that the data is stored in and it is configurable depending on the desired number of 9’s. Pulsar achieves durability using bookies running in storage nodes. When a Pulsar broker receives a message, it sends it to a set of bookie nodes. When message data is received by a bookie, it saves a copy in memory and also writes the data to a WAL. This log is forced to stable storage before the bookie sends an acknowledgment to the broker. Similar to database transactions, a WAL ensures that the data is not lost even if the machine fails and comes back again. By replaying the log, the machine will recover the data by replaying the WAL.

In addition to this, Pulsar messages can survive multiple node failures. Pulsar replicates each message to multiple bookies. After configured number of replicas have written the message to bookie successfully, Pulsar sends an acknowledgment to the producer. This guarantees that Pulsar ensures zero data loss even in the presence of multiple hardware failures.

Even with these strong durability guarantees, due to the BookKeeper internal IO implementation, Pulsar provides very high throughput and low latency.

Conclusion

Pulsar is a next generation pub/sub messaging system designed from ground up to address the gaps in open source messaging systems. In this first blog post, we highlighted the flexible messaging model generalizing both queuing and publish-subscribe in a single API. We discussed enterprise-grade features of multi-tenancy, geo-replication, and strong durability guarantees. In our next post, we will highlight additional features around I/O isolation between reads and writes, scalability, and security, in addition to operational maturity.

(Part 2 of this blog post is available at Why Apache Pulsar? Part 2)

This post features contributions from Matteo Merli and Karthik Ramasamy.

----------------------------------------------------
Thanks!
Matteo Merli