By Splunk December 11, 2007
I’m going over some stuff for the new support engineers, so I thought it would be useful to put it in a blog post. As an example of what you can do with conf files, I’ve got the changes I make to my own configuration and why. This is more focused on 3.1.x rather than preview, but I’m basically using the same configuration in both so far. For public consumption, I’ve changed some names but otherwise this is the contents of my conf files.
This first post is about inputs.conf, props.conf and transforms.conf, the basics of event handling.
inputs.conf
host = myhost [tail:///Library/Logs/CrashReporter] disabled = false sourcetype = crashreporter [tail:///Library/Logs/MySQL.log] disabled = false [tail:///Library/Logs/Software Update.log] disabled = false [tail:///Library/Logs/DirectoryService] disabled = false [tail:///var/log] disabled = false
I added the tail on /var/log from the UI but the rest of this I did by hand. That wasn’t strictly necessary, but it was easier for me to add a couple stanzas at once that way. “host = myhost” is setting the name of my machine so everything has the correct hostname even if something in the actual event might make it get set to something else. (syslog type events are the usual offender for me, even if I’m not actually getting syslog from another host. Some tend to show up as “www” if I’m not paying attention.) CrashReporter, MySQL.log, Software Update.log and DirectoryService are things specifically in /Library/Logs that I wanted. I needed to set the sourcetype manually for crashreporter, so I just listed the others while I was at it.
props.conf
[crashreporter] SHOULD_LINEMERGE = True BREAK_ONLY_BEFORE = ^Process: maxDist = 120 MAX_TIMESTAMP_LOOKAHEAD = 128 REPORT-mystuff = my-extractions SHOULD_LINEMERGE = False TIME_PREFIX = \[ KV_MODE = none sourcetype = my_access_log [osx_secure] REPORT-osx_secure = osx_secure-extractions KV_MODE = none
The crashreporter stanza here is why I manually set the sourcetype in inputs, I wanted to change the linebreaking. These are multi-line events all starting with “Process:” and if I didn’t manually configure this they might not be correctly recognized as all one event. We’ve gotten better about these things generally, but I wanted to make sure it worked the way I wanted. my_access_log and osx_secure are defining some custom field extractions. Most of my_access_log stanza is taken from the default access_common because I didn’t change the date format. The “KV_MODE = none” means only use the ones explicitly configured and don’t try to guess with any automatic key/value extraction. (That’s what “kv” means.) Because there’s no DEST_KEY set, these are search-time extractions and not actual indexed fields. Note that I’m manually setting the sourcetype for my_access_log here in props, so in inputs I could just tail the whole directory.
transforms.conf
[my-extractions] REGEX = (\S+) \[[^\]]*\] "(\S+) ((?:\w+://[^/\s]+)?(/[^\?/\s]*)[^\?\s]*?([^\?/\s]*))(?:\?\S*)? (\S+)" (\d+) (\S+)(? : "((http://[^/]+)?[^"]*)" "([^"]*)")? FORMAT = clientip::$1 method::$2 uri::$3 root::$4 file::$5 version::$6 status::$7 referrer::$8 referrer_domain::$10 useragent::$11 [osx_secure-extractions] REGEX = creating shared credential for user (\w+)$ FORMAT = login_user::$1
Here are the regexes defined for my fields. I took the default regex for access_common out of the defaults bundle and hacked it up for my custom log format. The FORMAT line is taking the various things in parens from the regex (identified in order by $1, $2, etc) and uses them to create the new extracted fields with the specified names. Note the clever use of the $ anchor so I could just pick off the last word from the osx_secure events.