By Splunk December 03, 2007
I have a lot of conversations lately about the topic of IT search versus SIEM (security information and event management), the more traditional way of doing security event management. People are asking me how Splunk’s technology is different from all the log management tools. With ArcSight (my former employer) going public, LogLogic going through some turmoil in their executive management, and Splunk that just got an amazing round of investment, people are very interested in understanding what the deal is.
The topics of SIEM and IT search are fairly similar. However, there are some very important differences that I want to start pointing out in a series of blog posts.
Let me start with the topic of data collection. In an SIEM system, you use a collector, a connector, or an agent (I don’t really care what you call it, but it’s some piece of code which reads the data and feeds it into the system) to process the data before you can use it in your SIEM for correlation, reporting, or forensic purposes. If you do not have a connector specifically written for your data source, you are out of luck. Just to be clear, I am not talking about having a connector for files, for ODBC, for SNMP traps, or for syslog over UDP/514. I am talking about a connector for each specific data source: Snort syslog, Snort database, CheckPoint OPSEC, CheckPoint syslog (do they have a syslog output?), PIX over syslog, CISCO router over syslog, etc.
What this means is that the SIEM has to either already support your data source or you need them to build you a connector; or you build it yourself. Most of the SIEM tools have some sort of an SDK that you can use for this purpose. However, do you have the man power and the skills in-house to do so? If not, does the SIEM company have the bandwidth to build your connector in acceptable time?
What happens, if the source data format changes? For example, Snort might slightly change its syslog format. Guess what has to happen. Yes! The connector needs to be updated to support the new format. This could mean a down-time of your data source of a few days, if you don’t plan accordingly and get an updated connector right away.
No connector – No data
Now, what is the deal in the IT search world? Well, you need some sort of connector as well. However, you only need one to transport the data from the data source into the search system. In other words, you need about a handful of connectors: One for ODBC, one for receiving syslog on UDP/514, one for text-files, and one for databases other than ODBC. (Okay, okay, I will add one for CheckPoint’s OPSEC). That’s it. You don’t need a specific connector for each data source. You also don’t have to update every time the data source decides to slightly change the logging format. [And if you think that never happens, have a look at SiteProtector.]
What does this mean? It means that from day one that you install your IT search technology, you are able to work with your logs. You don’t have to wait until the right connector is available.
So much for now. In my next post I will talk about structured data.
By Raffael Marty