Splunk for UNIX

Overview

This documentation applies to Splunk version 3.4.x and earlier ONLY

The Splunk for UNIX application supports a variety of different log records or data sources. Among them:

• Generic UNIX syslog events
• UNIX services, such as ssh, ftp, dhcp, cron, snmpd, named, and so on
• UNIX system monitoring (ps, top, vmstat, etc.)
• Linux iptables firewall
• OS X ipfw firewall
• OpenBSD PF firewall

The supported scripted inputs are:

• audit (rlog.sh)
• df
• iostat
• lsof
• netstat
• ps
• time/date
• top
• vmstat
• yum

Installation and configuration

See http://www.splunk.com/base/Documentation/latest/Admin/InstallSplunkApplications for generic instructions of how to install and configure this add-on.

Saved Searches

For all the searches, you should make sure that they are owned by the correct user. The easiest way to do this is to use the savedsearches.conf_local file and place it in $SPLUNK_HOME/etc/apps/local/savedsearches.conf. Edit the file and assign the searches to the right user and role (based on the template). Note, if you are using LDAP, you need to use user names, and not user IDs in the configuration file.

Scripted input configuration

To enable the scripted inputs to monitor ps, vmstat, iostat, and so on, do the following:

1. Modify any of the stanza's to tailor their enabled/disabled, interval ( sec ), source, sourcetype.
2. If you have installed in non-default places you may need to fix the paths in some of the bin scripts
3. By default all monitoring data is placed into the default index
4. The iostat input requires iostat installed. It's in the sysstat RPM which can often be installed with "yum install sysstat"
5. Enable the inputs you want in inputs.conf with "disabled = false"
6. Don't forget to restart splunk. Once running, check splunkd.log for errors.

iptables

For event types to work your logging should be prefixed with an action value that reflects the action taken by the firewall. This can be achieve with the following parameter --log-prefix "action=[PASS|BLOCK] " including the trailing space.

It's advisable that you log both INPUT and OUTPUT traffic. The example snippet below shows all outbound allowed traffic expect loopback being logged as well as all the rejected inbound traffic and allowed traffic to port 22, allowed inbound traffic to ports 8000 and 8089 as well as ICMP traffic is not being logged.

SAMPLE IPTables Configuration

From /etc/sysconfig/iptables: (on RedHat)

-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 8000 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 8089 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j LOG --log-prefix "action=ALLOW " 
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -j LOG --log-prefix "action=REJECT "
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -s 127.0.0.1 -d 127.0.0.1 -j ACCEPT
-A OUTPUT -m state --state NEW -j LOG --log-prefix "action=ALLOW "

auditd

auditd is available on Linux. To get audit data into Splunk use the rlog.sh input script by enabling it in the inputs.conf. Set disabled=false to do so. Here's how to set auditd up on a UNIX system:

Reference: http://www.cyberciti.biz/tips/linux-audit-files-to-see-who-made-changes-to-a-file.html

1. Install if needed, but it should be there

  yum install audit

2. Make sure it starts on boot

  chkconfig audit on

3. Start if necessary

  /etc/init.d/auditd start

4. Add a file to be watched

  auditctl -w /etc/passwd -p wa -k splunk

-w is file, -p is actions (write, append, read, execute), -k is an optional key tag added to the audit.log entry.

5. Make a change to the file

  useradd mrevil

6. see the change

  ausearch -i -f /etc/passwd

-i means translate so you see user's name instead of uid. -f is file to look for. Ausearch has lots of args for searching

SplunkLightForwarder and Transplant Parsing

If the Splunk for Unix application is installed on a forwarder where parsing is not done, such as the SplunkLightForwarder in Splunk versions 3.4 and later, or if an earlier version is manually configured for Transparant Forwarding, the Splunk for Unix application must be installed on the receiver as well.

This is because some of the configuration is used at data acquisition time, on the forwarder, and some at parse time, which occurs on the receiver in this type of configuration.

Operating system support

This application supports most UNIX flavors, such as FreeBSD, Solaris, AIX, Linux, and OSX

Tips and tricks

If you install this application on a lightweight forwarder, you will need to re-enable the exec processor. See instructions: http://www.splunk.com/doc/latest/admin/ForwardingandReceiving#Lightweightforwardingandrouting