Splunk at Yahoo!: Big Data at Scale

Update 9/27/16: As of Sept. 27, 2016, Hunk functionality has been incorporated into the Splunk Analytics for Hadoop Add-On and Splunk Enterprise versions 6.5 and later.

Big Data is a term that’s thrown around a lot by vendors, thought leaders and the press—so much so that it’s nearly lost all meaning. In fact, most people skip “big” and immediately discuss how it’s about more than just the amount of data (and it is). That said, we should take a moment to recognize what true big data scale means.

Today we announced that Yahoo is using Hunk to analyze 600 petabytes (yes, that’s a “p”) of data in Hadoop and is analyzing over 150 terabytes per day with Splunk Enterprise. That’s real scale, and Yahoo is using the Splunk platform to get there. But while the amount is interesting, what’s really compelling is how the company is using the data.

With Hunk, the company is tracking and improving the performance and stability of its grid system, and tracking the system metrics of all of its clusters. Yahoo uses analytics on Hadoop to visually browse complex tables, meet SLAs and gain insights into historical resources. By using Hunk, the company is saving millions of dollars per year in hardware provisioning alone.

Yahoo has also deployed Splunk Enterprise as its platform for machine data. Teams ranging from IT operations, infrastructure, products and security are using Splunk Enterprise to maximize revenue by understanding customer preferences, advertising and marketing campaign popularity, and click through rates, while also addressing IT workflow issues.

By implementing Hunk and Splunk Enterprise, Yahoo is pushing the boundaries of data analytics on the “big” side of big data, but in a way that’s also addressing core needs of the business. That’s the full power of big data—and the power of the Splunk platform.

Doug Merritt
Posted by

Doug Merritt

Doug Merritt has been Splunk’s President, CEO and a board member since 2015. He believes that data has the power to address many of the world’s most pressing problems, and that technology has the potential to enable anyone in any organization to become a data practitioner. Doug’s leadership focus is around driving large-scale, simultaneous transformations in high-growth environments. While at Splunk, he has led major shifts in the technology roadmap, financial model and go-to-market approach — dramatically increasing Splunk’s market capitalization. Splunk is one of the only B2B software companies to accomplish a transformation of this size and scope. Over his 30+ years in the tech industry, Doug has held senior leadership roles in organizations across a wide range of disciplines, including Product, Sales, Marketing and HR, for companies like Cisco, SAP and PeopleSoft. Doug holds a B.S. in business from The University of the Pacific and is an avid athlete.

Join the Discussion