I have recently been heads-down working on a large Splunk Cloud PoV (20+ TB / day), and the customer asked if Splunk supported their forwarding technology called Vector. I had never heard of Vector, so I took a note to do further research. I couldn’t find anyone else at Splunk who had seen this technology before, so I embarked on a little research project. What I discovered surprised me, Vector is actually fairly powerful. The best news is, Splunk works with this technology with ease! I want to share the results of this research with the rest of you!
This blog will detail what Vector is, performance statistics, advanced capabilities, and how to use it with Splunk. This way, you can be prepared if you run across this technology in the field. If you don’t care about the performance/capabilities and just need to learn how to get it up and running, feel free to skip to the bottom of the post.
What is Vector?
You can always read their documentation available here, but I will summarize it here in this post.
Vector is a new, open source, tool that lets you “take control of your observability data” written in Rust. Their main value propositions are:
- It’s Fast
- Vendor Neutral
- One Tool
- Handles All Data
- Programmable Data Transforms
- Clear Guarantees
Here is an architecture diagram showing how the Vector Agent fits into the Splunk platform. Note that Vector is shown forwarding data to Splunk Data Stream Processor. Vector can also forward directly to Splunk if needed.
It’s important not to confuse Vector with Splunk’s DSP, Vector’s their documentation clearly states:
- “You SHOULD NOT use Vector if you need an advanced distributed stream processing framework.”
- “You SHOULD NOT use Vector to replace Kafka. Vector is designed to work with Kafka!”
On the other hand:
- “You SHOULD use Vector to replace Logstash, Fluent*, Telegraf, Beats, or similar tools.”
Basically, you can think of Vector as an open source version of Splunk’s Universal Forwarder.
Why Is Vector Cool?
Here’s why it’s cool:
- The project is written in the Rust programming language. I could write an entire blog on why Rust is awesome, but here are the cliff notes:
- Rust is fast, like C++ fast
- Rust was built by the Mozilla Foundation
- Rust is open source
- StackOverflow polls show Rust is the most beloved programming language for 5 years in a row
- Rust is immune to the most common application security vulnerabilities with no performance penalty. (This is a bold claim, and my favorite feature of Rust, learn more about it here.)
- Vector is open source.
- It’s really fast!
- Their documentation rocks.
This is all great, but we don’t use software just because it’s cool. Performance in the field matters, and this is where Vector shines.
Performance and Capabilities
Vector’s claim to fame is that it’s the fastest forwarder on the market except in two categories:
- TCP To TCP forwarding, Splunk’s Universal Forwarder is still slightly faster here.
- Regex Parsing, FluentBit has the crown in this category.
Here are some data that show off the performance of TCP output and File To TCP performance. This is accessible on their main page.
Vector provides more detailed performance testing statistics on their GitHub account. Here is an excerpt:
$ bin/compare -t file_to_tcp_performance -c multi_file
| Metric | filebeat | fluentbit | fluentd | logstash | splunk_hf | splunk_uf | vector |
| Disk Thrpt (avg) | 7.8MiB/s | 35MiB/s | 26.1MiB/s | 3.1MiB/s | 39MiB/s | 40.1MiB/s | 76.7MiB/s W |
| CPU sys (max) | 4.7 | 4.5 W | 6.6 | 46.5 | 34.7 | 6.6 | 23 |
| CPU usr (max) | 63.8 | 50.3 | 50.5 | 98 | 84.7 | 18.1 W | 74.9 |
| Load 1m (avg) | 1 | 0.6 W | 0.6 | 2.2 | 2.6 | 0.6 | 1.5 |
| Mem used (max) | 170.7MiB W | 370.3MiB | 890.6MiB | 763.7MiB | 442.6MiB | 241.4MiB | 188.1MiB |
| Disk read (sum) | 466.9MiB | 2gib | 1.5gib | 187.3MiB W | 2.3gib | 2.4gib | 4.6gib |
| Disk writ (sum) | 177.6MiB | 6.1MiB W | 11.6MiB | 602.6MiB | 54.3MiB | 24.4MiB | 16.1MiB |
| Net recv (sum) | 240.6kib | 6.6MiB | 2.3MiB | 2.4MiB | 14.7MiB | 3MiB | 22.4MiB W |
| Net send (sum) | 167.3MiB | 2gib | 573.7MiB | 65.2MiB | 1007.4MiB | 1.2gib | 2.3gib |
| TCP estab (avg) | 183 | 1070 | 183 | 175 | 1358 | 122 | 186 |
| TCP sync (avg) | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| TCP close (avg) | 0 | 1 | 0 | 0 | 0 | 0 | 0 |
W = winner
filebeat = 7.1.1
fluentbit = 1.1.0
fluentd = 3.3.0-1
logstash = 7.0.1
splunk_heavy_forwarder = 7.2.6-c0bf0f679ce9
splunk_universal_forwarder = 220.127.116.11-962d9a8e1586
vector = 0.2.0
I am not going to go over all the performance metrics in this blog post, but I can summarize the overall results. In general, their claims seem to be true: Vector is impressively efficient with its CPU and memory use, and is faster than any other open source agent. If you are a performance testing geek like me, you can access all the raw results using this link. They detail the exact test cases, how the statistics were collected, system configuration, and related AWS information on where the tests were run.
Vector also lists advanced functionality provided by their tool. Of course, Splunk’s Universal Forwarder has all of this functionality as well, but it’s clear that the rest of the open source community is lagging behind in this category. Having these advanced capabilities puts Vector in a very strong position to be used with mission critical data that needs high reliability. The next screenshot shows a comparison of these advanced capabilities and can also be accessed on Vector’s home page:
It’s important to realize that these performance stats and capabilities not only increase the speed of a forwarding tier, it can also help organizations directly cut down on their cloud computing costs. For example, let’s pretend a company has built a log aggregation tier that uses Logstash to convert incoming TCP connections to HTTP. If this company replaced Logstash with Vector, then they can decrease AWS costs by up to 90% for their forwarding tier.
That all sounds great, so how do you install and configure Vector?
Installing Vector on my Mac was a breeze, and it’s similarly easy on any *nix-based system, and even Windows. Vector provides installation packages that support MacOS, Linux, Windows, and they also have a Docker image. Of course, because it’s open source, you can compile it to target other platforms as needed. I’m looking at the IoT and legacy OS folks here.
Note, these steps only work on Mac using Homebrew. I use Homebrew to manage all of my packages, and if you want to follow these steps, you will need this software. You can get it here.
Vector Deployment Steps
Step 1: Create a HEC endpoint in Splunk. If you don’t know how, here is a link.
Step 2: Install Vector (as mentioned before, this step only works on MacOS with brew installed, other installation guides can be found here)
brew tap timberio/brew && brew install vector
Step 3: Configure an input by editing the file /usr/local/var/lib/vector/vector.toml
# This configuration reads a single log file, and sends it to Splunk
# Input data.
type = "file"
include = ["/var/log/system.log"]
# Output data
inputs = ["in"]
type = "splunk_hec"
# Use the next line for on-prem deployments
host = "https://splunk-indexer:8088"
# Use the next line for Splunk Cloud deployments
# host = https://http-inputs-[your_splunk_cloud_stack].splunkcloud.com:443
token = "[your-splunk-hec-token]"
encoding = "json"
Step 4: Run Vector
brew services start vector
It’s really that simple, this configuration will immediately start forwarding logs to a Splunk instance. You can get it up and running in 5 minutes if you know what you are doing. This workflow should be familiar to Splunkers, as it’s very similar to the configuration of Splunk’s Universal Forwarder. I will admit, it took me a few hours to distill this information down, as I had some trial and error getting it to work with Splunk Cloud. This will be common knowledge for anyone that deals with HEC in Splunk Cloud, but it was new for me in this PoV. You can see that I have a commented line in Step 3 that shows the configuration if you want to connect the Vector forwarder to Splunk Cloud.
Vector’s website claims that the tool is downloaded over 100,000 per day. This means that we are likely going to run into this technology far more often in the future. While I don’t recommend replacing Splunk’s Universal Forwarders with Vector, if an organization is using any other forwarding technologies and doesn’t want to use the Splunk Universal Forwarder, then Vector is clearly the best play. Now you have enough information to start your journey with Vector and add value to organizations!