Forums: SplunkGeneral: New syslog agent

Greetings Splunkers and Splunkees,

I would like to introduce a piece of code that I've been working on recently that I think would be very beneficial to anyone that's implementing a centralized log architecture. It's called log minion and here are a few of the key features:

• It can "tail" any arbitrary log file on a system and send it across the network to a syslog server.
• It is capable of tailing multiple files simultaneously using a threaded algorithm
• It has full regular expression support for pre-filtering logs before sending them
• log levels, facility and application name are configurable
• It daemonizes itself on startup and can be run from a startup script.

you can download it from my website at:
http://mysfitt.net/projects/logminion.php

Please let me know what you think or if you find any bugs. I've been testing non-stop for the past two days but I'm only mortal ;-)

Thanks!
Joe Reeves

Joe,

Just downloaded it and checked it out.. very cool! Hopefully a lot of people will find this useful in conjunction with Splunk.

Thanks,
Brad

Thanks Brad!

I'm also working on another project that does exactly the opposite. It "pulls" the data instead of pushes it. So basically, you have it sitting on your Splunk server pulling files from remote systems. Database logs, apache access logs, windows application logs, anything that doesn't have a native remote transport builtin are all things that it can grab and drop directly in the Splunk "Blackhole" directory to be indexed.

The coolest part is that you don't even have to know what file transfer protocol to use. Brigand can auto detect what's available and use whatever user/pass credentials you have to get the file.

It's still very beta as I just wrote it over the weekend, but if anybody's interested, check out File Brigand:
http://www.mysfitt.net/projects/filebrigand.php

cheers :-)
Joe

Hi Joe,

I have tried to use logminion on a Solaris 9 but I get an error :
root@scnsys11:~/logminion/logMinion> ./minion.pl
This Perl has neither ithreads nor 5005threads at /opt/perl5.8.7/lib/5.8.7/Thread.pm line 335
BEGIN failed--compilation aborted at /opt/perl5.8.7/lib/5.8.7/Thread.pm line 335.
Compilation failed in require at ./minion.pl line 19.
BEGIN failed--compilation aborted at ./minion.pl line 19.

Any idea ?
Thanks for your help.
Pierre.

Joe,

It seems that I have to recompile Perl to have Thread support under Solaris.

Pierre,

thanks for the feedback. I could make a seperate build for Solaris without thread support, but it will lose the ability to tail multiple files at the same time. Do you feel that would be an acceptable solution?

Thanks,
Joe

Joe,

Thanks for your answer but we will try to compile perl with Thread support on Solaris. I think that the ability to tail multiple files at the same time is important. I will post a msg after testing.

Thanks.
Regards.
Pierre.

Usefool tool, but what I would like it to do is to tail a datebase log, like /var/log/app.20060516 and follow on to the next file when the day changes. Is this possible ?

Re: anonymous

Yes, Log Minion should be able to tail any type of file and send it to you syslog server. It is also designed to deal with rotated log files as it will block until it can read the file again instead of exiting. I guess I should make a note of that somewhere in the docs or description :-) As long as the file being written to keeps the same name, you should be fine. Note that Minion is designed to create a new syslog line for each log line that is terminated by a line end. So if your logfile is in a binary format instead of text format, it probably won't work as desired. It might be possible for me to work some binary support into a future release though if there's sufficient interest in that feature.