Digital Resilience Pays Off
Download this e-book to learn about the role of Digital Resilience across enterprises.
Whether it’s familiar data-driven tech giants or hundred-year-old companies that are adapting to the new world of real-time data, organizations are increasingly building their data pipelines with Apache Kafka. Confluent has an impressive catalog of these use cases.
Splunk Connect for Kafka introduces a scalable approach to tap into the growing volume of data flowing into Kafka. With the largest Kafka clusters processing over one trillion messages per day and Splunk deployments reaching petabytes ingested per day, this scalability is critical.
Overall, the new connector provides:
As a sink connector, Splunk Connect for Kafka takes advantage of the Kafka Connect framework to horizontally scale workers to push data from Kafka topics to Splunk Enterprise or Splunk Cloud. Once the plugin is installed and configured on a Kafka Connect cluster, new tasks will run to consume records from the selected Kafka topics and send them to your Splunk indexers through HTTP Event Collector, either directly or through a load balancer.
Key configurations include:
To get started, download Splunk Connect for Kafka from Splunkbase. Install this JAR file across your Kafka Connect nodes, and restart your these nodes with updated properties to enable the Splunk connector. Please consult our documentation for additional instructions, configuration options, and troubleshooting. Also, check out Don Tregonning’s blog post for a full guide to getting set up and tips for configuring your end-to-end pipeline.
With the introduction of Splunk Connect for Kafka, we recommend shifting existing use of the legacy Splunk Add-on for Kafka for consuming Kafka topics to the new connector. The add-on will continue to be supported for monitoring your Kafka environment using JMX.
And in case if you are wondering, yes—Splunk Connect for Kafka is open source! You can access the source code at our github repo.