Forums: SplunkGeneral: Why Splunk at all?

I mean the interface is cute and all, but why not just use a LogWatch or LogCheck script -- they mail me when there's an issue that needs to be looked into. Am I missing something?

I think the nice thing about splunk is that it's a front end that you can use when things go wrong. Sure, it's nice to use stuff like LogWatch to look for events that you already know about, but what do you do when the world comes crashing down around you? If you haven't already configured LogWatch to detect the particular problem you're having, then the best alternative you have is to start grepping through log files. Frankly, Splunk is WAY faster and more effective for tracking down new, unknown problems.

It's also useful when your manager comes up and says something like "How many of our users are using sudo to run thus-n-such?" Our syslog data is over 1GB per day, and using grep to find this kind of information over the course of a week takes over an hour. Splunk can do the same thing in about 20 seconds.

So why splunk? Because it works!

I'll also chime in that Splunk Professional (and maybe the free version?) has LogWatch-like features where you can search for a particular pattern/event and have it e-mail you in certain situations (too many messages, not enough messages, messages matching this-or-that)

We're not planning to use Splunk at all for this; we've got our own in-house solution that monitors log traffic and alerts us to events. However, many members of my team thought it would be cool to be able to set up custom alerts for trivial things such as debugging a problem or a piece of software.

The fact is that Splunk is very flexible and can add a lot of value to a centralized log collection server. Much more than "simple" tools like grep, awk, sed, and swatch.

Hi Folks,

I think Paul has hit the nail on the head here. While splunk in the professional version
is capable of running searches on logs to look for common errors it's not meant as a
replacement to something like LogWatch. LogWatch and other products like it are setup
to look for known problems in the logs and they do a very good job at it. Splunk's usefulness
is in dealing with problems that you haven't seen before, or for looking for systematic
problems in the system by being able to search though multiple log files that are connected
to each other in some way ( appserver --> Db logs ).

Any time a problem occurs that is unforseen I see most sysadmins check logWatch or nagios or
the like first to see if it's an obvious problem. When that fails the next option is to start
reading log files, If they are on one server thats fine but when you have them spread across
several servers then the problem gets a little tedious and it is here that splunk will prove
the most useful.

The monitoring stuff is a nice add-on but it is in no-way meant to compete with something like
LogWatch which looks for patterns in the logs, we index the log and add meta data, not an
efficient way if you're just looking for known patterns but when you need it to search for
problem to which there is no known pattern then you need the log chopped up and sliced to allow
the sysadmin navigate it in a way that is more efficient than grep or awk. Trying to grep through
multi-line logs like websphere is pretty painful because of multiline stack traces. Now you can
tweak your grep scripts to handle this but it's never going to be a nice as having the data all
in one search interface, and when you get new log formats you have to change your scripts to
add those types in, with splunk it will type the file and figure out if it's multiline all on it's own.

We are not looking to replace some of the very good monitoring tools out there, we use loads of
them there in the office to keep tabs on our network. From my perspective, as an engineer working
in splunk, we are trying to make the time to track down a new problem a little faster. It's not
a magic bullet but hopefully it will allow sysadmins to find new problems faster as they crop up,
and when your boss is in you cube asking why a server is not working getting it fixed 5 mnins
earlier and getting him outta your cube a little earlier is worth alot.

cheers,
Rory

I have been in the process of implementing a series of searches that have the same excludes as logwatch, so far this is a big project, but not entirely undo-able. It seems to me that the next logical step for splunk would be this. Why would I want 500 logwatch messages every day, when I could have a summary in one email?