Forums: SplunkGeneral: Sucking config files in

I'm trying to suck configuration files into splunk. I want to do this with the batchloader. The batchloader would monitor all changes on config files.

My splunk configuration looks like this:

props.conf:
[hostsfile]
DATETIME_CONFIG = CURRENT
SHOULD_LINEMERGE = True
AUTO_LINEMERGE = False

How can I get it to work?

BTW: This is on Splunk 3.0B3

Good question.
We here at splunk do this all the time.

If you tail /etc it should do the right thing unless you have created your own source types for those files.

In our default bundle SPLUNK_HOME/etc/bundles/defult/props.conf it defines the following:
If you look in this file you will see:

[source::.../etc/...]
sourcetype = config

[config]
BREAK_ONLY_BEFORE=goblygook
MAX_EVENTS=200000
DATETIME_CONFIG = NONE
CHECK_METHOD = modtime
pulldown_type = true
category = Config

The top stanza says that any file that we eat in /etc... will be source typed as config.
The second stanza says that anything that is sourcetype config to - break only when it sees goblygook, max size of any event 200k lines, that the time stamps come from mod date of file, that is a pulldown type, etc.

Can you tell me if you've manually set the files your eating to be another source type?
If not, it should just work.
If you have, then you can apply the above config to your soucetype and it should work.

The nice thing is that onces it works, splunk can be used to keep track of changes to your configs.
Every time the config changes splunk will re-index it.
You can do things like search for:
source::/etc/syslog.conf
It will show an "event" every time the file has changed.
If your eating configs from multiple hosts you can event make source they are all the same file.
source::/etc/syslog.conf | diff

Let us know if this does not work as expected/explained

Thanks for your answer. It seems that in 3.0 there are a lot of new properties available...

There is one problem in b3. The default bundle names the sourcetype as config, whereas the config bundles calls it config_file.

It seems that there are some new attributes that are not documented yet (pulldown_type, category, check_method). Can you shortly explain what they mean?

Thanks...

Now that I played a little bit with this feature I found two little glitches...

I tailed /etc/hosts into splunk and added after a while a hash (#) to the end of the file. This worked great. The file was completly re-read. But when I added a comment behind the hash, a new event was generated, with only the comment in it. It seems that splunk is in some cases tailing the file, instead of re-reading it from the beginning

The second glitch is the eventtime. I would assume that the file modification time would be used as the eventtime. But this does not seem to be the case. This would be an important feature, for monitoring.

Hum, good stuff.

To fix the first problem, set
CHECK_METHOD = modtime

for the timestamp issue, set the following to force the system to use modtimes.
DATETIME_CONFIG = NONE

I'll update the stand configuration for ect to do this.

Thanks, and i'll add some test cases to our automated framework to cover these cases.

Cheers,
e

The settings under the config bundle already contain the CHECK_METHOD and DATETIME_CONFIG settings. There must me an issue elsewhere.

I guess in the first case, this happens, when changing files too fast.

In the later case I think it uses the indexing time despite of the DATETIME_CONFIG...

Let me look more into it - busy day but will post something by eod.
e

Hey Burana400,

I believe if you try CHECK_METHOD = entireMD5 you'll have much better sucess.
The modtime just lets the system know when to look at the file again. IF your
config file is longer than the standard crc then if you add to the end of the file
it will just index the new data added to the file. If it crc's the entire file
then any small change to anywhere in the file will cause the system to eat the
whole thing again. This config parameter is specifically for config files
that you want to eat the entire thing everytime a change occurs to it.

cheers,
rory

The entireMD5 sound logical, but it doesn't work.

-I added a hash to the end of the file => an event was created with just the hash.
-I added "blabla" after the hash => an event with only "blabla" was created
-I removed the last line from the file saying "# blabla" => an event was created for the whole file

After that I added 5 characters somewhere in the middle of the file. => an event was created with the last 5 characters of the file.

Are you sure that tailing is the correct way to do this?

Also, the event's time does not match the file modification time...

ugh this doesn't sound good, any chance I could get a sample of one of these config files. I'll
take them and try and see whats going on. From the sounds of it you have the correct
config here so I want to confirm that this isn't a bug, don't want you chasing your tail
when the software is the one causing the problem.

You can sent it onto support, let them know I asked for it.

cheers,
Rory