By Splunk July 17, 2008
What do you do if your system administrator locks you out of your critical systems, changes the root password and then quits? If you haven’t thought about this, you are not the only one. San Francisco officials are facing exactly that question. A disgruntled employee locked out all the system administrators from some fairly critical systems, as you can read in the San Francisco Chronicle.
Insider crime is an area in computer security that still doesn’t get much attention. One of the problems is that the frequency of incidents is fairly low and therefore the problem rates low on a company’s charter. However, the big problem is that the average cost of such an incident is really high. In reality, companies are still struggling with protecting their perimeter. They are worried about outside attackers, script kiddies, about their competition breaking in, attacks of Chinese hackers, Russian crime rings, etc. They should balance their efforts to protect from these threats as well as from malicious insiders.
In this specific case, there were some very obvious signs that should have been noticed. The employee should have been on a watch-list and his activity should have been under review. He was about to be fired. This should have put him into a group of people that are monitored closely. Monitoring is not easy. It is all about people and processes and a little bit about technology. There is unfortunately no software or security tool out there that could detect an insider. And there will never be one.
As I point out in my book, you need to define a process that classifies employees. People on a watch list need to be monitored more closely. Audit records need to be recorded, especially for privileged activities (such as the ones executed by system administrators). Those records then need to be stored in a place where nobody can tamper with them (for example in Splunk). The records then need to be reviewed on a regular basis. Hopefully by a separate team. Ideally the reviews are automated to ease the work load (for example through alerts in Splunk).
A second step has to be the implementation of proper security processes. Separation of duties, for example. The system administrator by himself should not be able to alter all the passwords necessary to access a system. In reality, this is really hard to enforce. However, if the preventative control cannot be enforced, a detective control should be put in place. Firstly, system logs should be centrally collected and analyzed, and secondly, the file systems should be monitored for changes. That way, all changes can be reviewed to see what the exact impact of Terry’s actions was.
Traditional computer security attacks are violating policy. Specialized sensors can be developed and deployed to monitor for signs of attacks. Insider crime is often executed without violating any policy. For example, a system administrator has the right to change passwords. However, as in San Francisco’s case, Terry abused that privilege to lock everybody out of the machines. The net is that one has to monitor not just violations or obvious attacks, but also regular and seemingly benign activity. This results in a huge amount of data from a lot of different sources. Make sure you have a solution that can deal with all of it.
An Interesting side fact: The department of technology is worried about a third-party accessing the systems with Terry’s account. This is definitely the time where Splunk needs to be in place to monitor all the records to check for any account access. This information can then be used by law enforcement to take action.
This article: “San Francisco Hack: Where Was the Oversight?” contains some of my comments about the case.
By Raffael Marty