By Splunk December 18, 2017
Santa’s been having a rough time of it in the past century. First there was the giant North Pole ice storm of 1939, which nearly prevented him from delivering any presents. Fortunately, that was also the year that one tiny reindeer with a glowing red nose saved the day.
More recently, in the last 40 years, the global population explosion has started to put a strain on his traditional systems, not to mention the little mishap of spilling Mrs Claus's scalding hot chocolate onto his precious list back in 1972. Then in the last decade he’s faced some serious competition from upstart technology companies that promise to process orders and deliver them almost as fast as he can!
So, this year, 2017, just mere weeks before the holiday season was to start in earnest Santa decided it was time for a change. He called his Chief Technical Elf into his cozy cabin for a serious chat.
“Brains,” he said, “Things have got to change. The old ways have served us well for generations and generations, but I feel as though they are beginning to fail us now. We are receiving more emails than ever before! Children don’t write letters anymore. Also, there aren’t just ten or so toys to choose from, there are hundreds and hundreds! I’ve hired more postmaster elves to sort through the emails and process all the requests, but they can’t keep up. I need a new system that can ensure that is fast and scalable. Can you build it for me?”
“Well...of course I can,” answered Brains. “We’ll start investigating technologies right now that fit your requirements.”
Once Brains had processed Santa’s requirements he and a couple of developer elves started to research technologies that could help them design and build a scalable system quickly. Brains knew that due to the time constraints he would need to choose the most architecturally simple solution to service the requirements. Scale, flexibility and low latency were absolutely essential. He could tap one or two elves from the development team, but couldn’t risk pulling too many off their current duties. He began to do some searches online and very quickly began to eliminate possible technologies because they couldn’t service requirements. However, before long he settled on a choice.
After deciding that Apache Pulsar was the unified queuing and streaming solution they needed it was time build the application. In just a couple of days they had a working prototype ready for the big boss.
Watch the video below and then grab the code and try it for yours(elf)!
The code for this project is on GitHub: https://github.com/streamlio/santasaver.
$ git clone https://github.com/streamlio/santasaver
This example consists of three modules:
- An incoming email generator which is implemented by the incoming_emails.py file. It simulates emails coming from all over the world by generating a random tuple consisting of a neighborhood ZIP code and desired toy.
- PostMaster which routes the incoming emails generated above into its appropriate category of the toy. This is implemented by postmaster.py
- Lego Elf is the elf that manufactures lego toys. It reads from the lego category populated by the postmaster and after creating the lego, puts it into the correct neighborhood. The code for the modules is found in src directory. The resources consist of all the resource files needed by the Incoming email generator. The conf file is a log4j log config used by both the PostMaster and the Lego Elf.
More on Apache Pulsar
If you want to learn more about Apache Pulsar, please visit the official website at https://pulsar.apache.org.
You can also participate in the Pulsar community via:
- The Pulsar slack channel. You can self-register at https://apache-pulsar.herokuapp.com.
- The Pulsar email list.
Happy Holidays from all of us here at Streamlio!
----------------------------------------------------
Thanks!
Sanjeev Kulkarni