PowerShell Resource Kit for Splunk

Earlier this week Splunk launched a preview of the PowerShell Resource Kit on Github. It is designed to help Splunk admins who want to automate administration of their Splunk deployments, but also has some pretty cool recipes for integrating Active Directory with Splunk, forwarding alerts and other data to Microsoft’s System Center Operations Manager, and performing a rudimentary topology discovery via an automated Splunk search with PowerShell.

Many of Splunk’s customers monitor very large Windows environments that have hundreds if not thousands of Universal Forwarders. They also use Active Directory for authentication, policy management and general asset organization. It turns out that they also use PowerShell extensively. So, why not develop a Splunk module for PowerShell?

The PowerShell Resource Kit for Splunk has five main components:

Check and Manage Splunk Services
Active Directory tends to be at the center for most Windows environments, so the Reskit takes advantage of AD’s position by querying AD for a list of computers and determining which ones have Splunk services running on them. From here, you can determine the status of Splunk services and stop/start/restart Splunk services. This feature can be very helpful in determining if the box is having trouble or if Splunk, itself, is causing problems.

Search Splunk
There are several interesting use cases for executing a Splunk search with PowerShell. One of the more interesting ones is to search for specific workloads on a machine to determine if Splunk should collect data for predefined scenarios. This loose form of introspection can make it much easier to enrich a data source with details about the nature of its data.

Manage Splunk Server Classes
There are a variety of things you can do for Server Classes with the Resource Kit. As with managing Splunk services, you can get a list of all the server classes, as well as their associated apps and matching patterns. This granularity allows you to filter via whitelists or blacklists or even determine which deployment clients are connected. The Reskit can also create or remove server classes. And if you use Active Directory, the Reskit can ‘discover’ various server classes running on hosts (or not) so that you can either manage them or deploy them.

Configure Multiple Forwarders
Some would say that configuring multiple forwarders is the primary value add of the PowerShell Resource Kit. If you have multiple forwarders, configuring and managing each one at a time is not only tedious but error prone. As any good scripting language, PowerShell can automate this process for as many forwarders as you have.

The Reskit can apply input and output configurations to multiple forwarders at varying tiers in Active Directory. You can apply a specific input/output configuration to every host in an AD site or in a specific object unit. You can also install applications to multiple forwarders across all hosts at these varying AD tiers. Once installed, you can then apply the app configuration to all those forwarders with the app.

Run Tasks that Impact Multiple Hosts in Parallel
Another key value add that is implicitly supported by PowerShell is the ability to run tasks that can be applied to multiple hosts. Tasks, such as configuration updates and restarts/shutdowns, are common and tedious without an automation tool. They can even become inhibitors if you have a large environment that requires constant updates.

Forward Splunk Alerts to System Center Operations Manager
As a prelude to future versions of the Resource Kit, we decided to add one integration module that should pique the interests of SCOM users. We have received many requests for a Splunk/SCOM bi-directional integration, but had only achieved half of the requirement with a SCOM app that resides on SplunkBase. This app allows you to right-click on an alert in SCOM to launch Splunk with specific searches related to that alert.

Although it is an interesting scenario, most people want to send alerts generated by Splunk to SCOM for alert management. This Reskit recipe allows you to do that by creating an SCOM monitor based on a Windows PowerShell script at the following URL:
• http://technet.microsoft.com/en-us/library/ff381420.aspx

Modify the PowerShell script in the example to pull data from your Splunk indexer. In this example, a hard search is made against the Splunk indexer to find any instances of the string “error” found in the last five minutes; the SCOM monitor reports a bad result if any instance of the string “error” is found in the Splunk indexes.

param($computerName)

import-module Splunk;
$credential = c:\tools\load-credentials.ps1;
connect-splunk -computername $computerName `
-protocol ‘https’ `
-port 8089 `
-credentials $credential;

$fiveMinutesAgo = ( get-date ).addMinutes( -5 ).toString( ‘s’ );
$results = search-splunk –search ‘error’ –starttime $fiveMinutesAgo
$testSuccessful = -not $results;

$api = new-object -comObject ‘MOM.ScriptAPI’
$api.LogScriptEvent(‘MyScript.ps1’,20,4,$computerName)
$bag = $api.CreatePropertyBag()
$bag.AddValue(‘ComputerName’,$computerName)
if ($testSuccessful -eq $true)
{$bag.AddValue(‘Result’,’Good’)}
else
{$bag.AddValue(‘Result’,’Bad’)}

$bag

We would love your feedback!! Please send any and all feedback about the Reskit to bbrauer@splunk.com.