Splunk Investigate was created with the core idea of providing modern app-dev teams with a collaborative investigative solution to accelerate remediation and resolution. By taking you through the use-case of monitoring your own computer with Splunk Investigate, we will show you how to get started! By the end, you’ll be ready to add the rest of your infrastructure (or maybe just your friends’ and coworkers’ computers!) and dive deeper into analyzing all of that data.
Steps we’ll be completing today:
- Set up your Investigate tenant
- Create a data pipeline (scripted)
- Configure a Splunk Universal Forwarder to send data to your Investigate tenant (scripted)
- Create a workbook that monitors your computer and collaborate with a friend
To start, you’ll need a Splunk Investigate tenant - this is your area where you can set up data transformations, store your data, search it and create workbooks and dashboards. Head over to splunk.com/investigate and register, and check out Splunk Docs for a more detailed overview of Splunk Investigate.
Once you have access to a Splunk Investigate tenant, you’ll want to get your data into it. For this post, we’re providing scripts to breeze over this part. In short, the scripts are going to:
- Create a data pipeline in your Investigate tenant to handle the data we’ll be sending it
- Configure the tenant to receive data
- Configure the Splunk Universal Forwarder to send logs to your tenant
Run this command in your terminal:
(Note: this command is only for Mac OS X. Support for Windows and Linux is forthcoming.)
export RECEIVER_PORT=9997 \ && export INSTALL_LOCATION=/Applications/ \ && export LOG_SOURCES=\$SPLUNK_HOME/var/log/splunk/*.log*%uf,/var/log/system.log%systemlog \ && curl -ksL -o osx-si-agent.tgz https://si-investigate-your-local-machine.s3.us-east-2.amazonaws.com/osx-si-agent.tgz && tar -xzf osx-si-agent.tgz \ && cd osx-si-agent \ && bash configure_si.sh \ && bash install_uf.sh
Note: This is an interactive command and will require you to answer some questions, as well as enter your username and password for your Splunk Investigate account.
Now we’re ready to see our data! Log into your Investigate tenant (https://si.scp.splunk.com), and click on “Workbooks” at the top.
Click on “Create Workbook” in the upper right.
Then click “Search” on the right.
You should now see a new untitled workbook that looks similar to this:
Let’s start with seeing our raw data that’s getting sent in. Enter the below in the search box and press enter:
| from main
You should see records returned, with the various system statuses that are getting reported from your computer.
Let’s focus on just the system logs by using a “where” filter:
| from main where source = “/var/log/system.log”
The results you see now are only the logs from your /var/log/system.log file on your computer
While this data is useful on its own to scroll through, let’s go ahead and build on this search by adding a child search. On the right is the child search icon, click this and you’ll see a child search generated beneath your current search panel.
In this child search, data from the parent search gets fed into it. This workbook flow allows us to build our searches incrementally, and even branch them when we want to have different analysis paths.
Let’s use this child search to look for messages reporting a failure:
| search “failed”
We now see only the log messages that contain “failed” somewhere in them. Further analysis could be done by creating more child searches from this one. Or you could go back to the initial parent search and create another child search to explore another set of logs from the /var/log/system.log file.
As you Investigate your data more in the workbook flow, it’s good to leave notes so you remember why you ran a search the way you did. You can add comments to your searches via this button:
You now have a workbook where you can explore stats about your computer! Where to go next? There are a few other tutorials available within Investigate that walk through other datasets. You can also invite coworkers or friends to your tenant to collaboratively explore data. And add more data sources — other computers, file sources, etc.
Go forth and Investigate!