Splunk Field Hashing & Masking Capabilities for Compliance

The release of Splunk 9.x includes exciting new features that make it easier to mask, hash, and filter data on disk and in the UI. In practice, this means you can satisfy various internal and external compliance requirements using Splunk standard components. Examples of compliance mandates include GDPR, PCI, HIPAA and other general-purpose PII and PHI protection mandates. In the past, you had to rely on third-party data pipelines, tokenization software, or scheduled summary searches in Splunk to achieve the same outcome.

The features I’m referring to are ingest actions and role-based field filtering (preview), and we will describe both in this blog.

Ingest Actions

The ingest actions feature is a data pipeline that runs on the indexer, heavy forwarder, or in Splunk Cloud. It executes before the data is written to disk. You can mask, filter, and route the data to one or more destinations, including AWS S3 or another Splunk index.

In the example below, we have data from Office 365 OneDrive where the events show an employee opening a file. We would like to mask the employee ID accessing the file and send this redacted event to another index.

^{The ingest actions interactive GUI displays the changes applied to data as it moves through the pipeline.}

The outcome is that for each event that comes in, an identical event where the UserId field has been masked irreversibly with “xxxxxx” is sent to the index “o365_redacted”.

^{The search interface shows duplicate events, one original and one redacted.}

Once you have the redacted data in another index, it is easy to apply role-based access control to the unredacted data.

Pros

• Easy and intuitive to use.
• Maintains compliance with “raw” logging mandates.
• Easy to set up role-based access control based on redacted indexes.

Cons

• You are duplicating the data (and doubling the storage requirements), making it less ideal for really large datasets.
• Currently no tokenization or hashing, only masking of data.

Role-Based Field Filtering

Role-based field filtering is applied to events at search time in the GUI only, as opposed to the data written to disk. This capability lets you limit access to confidential information for certain roles by redacting or obfuscating fields in events within searches without removing data from the backend.

Field filters retain the original event, but remove specific indexed or default fields from search results, or replace specific indexed or default field values at search time when those fields appear in the results.

Role-based field filtering is available in public preview for Splunk Enterprise 9.x and later, and by request for Splunk Cloud Platform customers. GUI support is not currently available to configure this feature, so you must modify limits.conf and authorize.conf to enable and configure it.

SPLUNK_HOME/etc/local/limits.conf
[search]
role_based_field_filtering = true

For example, to imitate what we did using Ingest Actions earlier, we can apply the following configuration to authorize.conf to filter data for a user role. The data format is JSON.

SPLUNK_HOME/etc/local/authorize.conf
[role_redacted]
fieldFilter-_raw = s/"UserId": "[^\"]*/"UserId": "xxxx/

As you can see, we are using a standard regex sed command to replace a field value with “xxxx”.

^{Screenshot from the redacted raw event for the user role where the role-based field filtering configuration is applied.}

^{Screenshot from an admin role where the unredacted data is shown.}

With role-based field filtering you can also apply hashing to indexed fields to perform analytics on the data–the UserId field in this case.

The additional configuration will look like this:

SPLUNK_HOME/etc/local/authorize.conf
[role_redacted]
fieldFilter-UserId = SHA256
fieldFilter-_raw = s/"UserId": "[^\"]*/"UserId": "xxxx/

The resulting search experience will display the redacted _raw field as “xxxx”, while the UserId field, which is an indexed field, will be replaced by the SHA256 hash value.

^{Screenshot from the redacted raw event and the SHA256 hash of the UserId field for the user role where the role-based field filtering configuration is applied.}

Pros

• Maintains compliance with “raw” logging mandates.
• No duplication of data.

Cons

• No UI for configuration (yet).
• Will not work with tstats, mstats or datamodel commands.
• Field hashing only applies to indexed fields.

Role-based field filtering is available in public preview for Splunk Enterprise 9.x and we are currently incorporating the customer feedback we are receiving during this preview.

Don’t just take it from me, try out these great new capabilities in Splunk Cloud Platform and Splunk Enterprise.

Happy Splunking!

Authors and Contributors: As always, security at Splunk is a family business. Credit to authors and collaborators: Johan Bjerke, Audra Streetman, Mikael Bjerkeland

_{Image source: “Log file where some text is replaced with xxxx, digital art” by DALL·E}