By Splunk March 16, 2016
I had a question this morning from a customer who was looking for ways to monitor changes made to PowerShell scripts in their environment. They wanted to know who made the changes, but also what changes were made. Well, I thought to myself–that’s a great excuse for a blog post!
Let’s break this down into two separate requirements:
- I want to know when a PowerShell script has been modified
- I want to know the changes between two versions of a file that has been modified
Who changed a file and when?
Requirement #1 is not hard to do using Splunk in combination with some native Windows file auditing features. In fact, it’s such a common use case that we’ve documented all the steps in the core docs. In short, this technique involves: in Windows, enabling NTFS filesystem auditing for the path in question; and then in Splunk, enabling security event log monitoring.
The end result will be audit events in Splunk that will look similar to the below. (Sorry, I could only find a group policy change quickly; a filesystem audit event will have a few differences.)
The lower body of this event even shows who made the change, for example:
Subject:
Security ID: SEATTLE\Administrator
Account Name: Administrator
Account Domain: SEATTLE
Logon ID: 0x55e65c847
Of course, you can create an alert or make a dashboard or report to then surface these events up to those who may need to take action.
But *what* changed?
Now we come to the reason that I wanted to break this into two separate requirements. I think that a best practice for authoring scripts used in production scenarios is to check them into a source code repository like git. That may still be a new concept to many sysadmins, but I hope that this changes, because there are many excellent reasons for admins to use some tools which used to be only for developers. (And hey, that’s what DevOps is all about–one of my favorite topics.) Some reasons for using source control for config files and scripts include:
- track changes in files over time
- rich “diff” tools to visualize these changes
- package an approved set of changes as a “release”, maybe even tag the release with a change control ID if you’re into that
- source control systems produce data that can be Splunked!
Speaking of visualizing diffs, here’s a quick screenshot of viewing a change in my favorite editor, Visual Studio Code (which is great for editing PowerShell files, by the way). Here is the list of changes in my project:
And here’s the change in detail, highlighting the addition of a paragraph:
You might be thinking by now, “ok, looks cool, but show me how to do this in a Windows environment with PowerShell scripts”. To answer that, I’m going to send you to a fellow PowerShell MVP named Stephane van Gulick. Stephane has written a comprehensive post on this very topic that I liked: Embrace version control using powershell git and github.
What’s next?
There are some related topics that I wanted to mention before letting you go. If you go back to the original question, one assumption that came to my mind was that this was being done not just for auditing purposes, but also for security and stability. In that case, what you really want to do is prevent all modified scripts from being executed, thus preventing any damage which might occur. PowerShell has a feature for this called the Execution Policy. Be sure to research how you can sign PowerShell scripts, and going one step further, prevent scripts that aren’t signed, or that have an invalid signature, from being executed. Or maybe you just want to verify the signatures, and for auditing purposes you might Splunk the output of that check.
And lastly, I should mention that you ought to follow such changes as I’ve described above with policy and training. Once everyone is on the source control train, it will be much easier to have quality discussion around a change control process, and how it can contribute to a tighter IT shop all around.
----------------------------------------------------
Thanks!
Hal Rottenberg