By Splunk June 23, 2008
Pretty much exactly 5 years ago, in June 2003, Gartner declared Intrusion Detection Systems to be dead. Before Gartner can do so, I will state that SIM is dead.
The crime landscape has shifted. We used to be worried about network layer attacks, TCP/IP attacks where funky flags were crashing your systems. This is gone. We really don’t worry about them anymore. We have systems to stop these attacks. The crime has shifted up to the application layer. There are attacks over instant messaging, there are SQL injections, there are application layer attacks. You have to start monitoring the application layer. Compliance requirements are shifting too. For example, the PCI DSS 1.1 requires the usage of application layer firewalls by June 2008. Applications need to be verified for vulnerabilities and not just the platform.
Some of the problems I see with Security Information Management are (the first four are adapted from the Gartner IDS press release):
- False positives in correlation rules
- Burden on the IS organization by requiring full-time monitoring
- A taxing incident-response process
- An inability to monitor events at rates greater than 10.000 events per second
- High cost of maintaining and build new adapters
- Complexity of modeling environment
However, the biggest problem lies in the fixed event schema. SIMs were built for network-based attacks. They are good at dealing with firewall, IDS, and maybe vulnerability data. Their database schema is built for that. So are the correlation rules. Moving outside of that realm into application layer data and other types of logs can get hard. Fields don’t match up anymore and the pre-built correlation rules don’t fit either.
We need a new approach. We need an approach that can deal with all kinds of data. An approach that deals with multi-line messages, with any type of fields, even with entire files as entities. There is a need for a system that can collect data at rates of 100.000 events a second and still perform data analysis. It needs to support large quantities of analytical rules, not just a limited set. The system needs to be easy to use and absorb knowledge from the users.
The solution is called IT search.