Audit Logging: A Comprehensive Guide

Audit logging, or audit trails, answer a simple question: who did what, where, and when?

So, in this article, we’ll answer our simple question: How can you use audit logs, and what use cases do audit logs best support?

What are audit logs?

When you use a technology service or product, audit logs are generated in response to every user action and system response. These logs capture critical information that can be used to:

• Authenticate users.
• Identify and validate requests.
• Route requests to the appropriate service nodes.
• Perform relevant technology operations and processing.

Difference between audit logs and regular system logs

While both audit logs and system logs record events and actions, they serve distinct purposes:

Audit Logs capture who did what, where, and when. They are primarily used for compliance, security, and computer forensic investigations. Audit logs track user actions and system changes to ensure accountability and traceability. They provide a chronological record of activities, crucial for audits and compliance checks.

System Logs primarily record system events and operational activities, such as errors, performance data, and service statuses. System logs are mainly used for debugging, monitoring system health, and optimizing performance. They offer insights into the operational state and efficiency of the system.

(Log data 101: what log data is & why it matters.)

Why is audit logging important?

Though the micro-actions behind audit logs are essential, the broader purpose of audit logging is even more significant. The main objectives of collecting audit logs are two-fold:

• To identify errors and improve system accuracy.
• To understand the intent behind activities, which can be used later for accountability or compliance.

At every step, the systems generate and record a trail of log and metrics data or metadata. This documentation can be utilized for various use cases, including security, monitoring, performance analysis, and cyber forensics.

(Related reading: log aggregation, log management & MELT: metrics, events, logs, traces.)

Roles and restrictions for viewing audit logs

Access to audit logs is typically controlled based on user roles within an organization. Different roles have varying levels of access and permissions to ensure security and compliance. Common roles and their associated access levels include:

• System administrators: These users usually have full access to all audit logs. They are responsible for managing and maintaining the system, ensuring that all activities are properly logged and monitored.
• Security officers: These users have access to audit logs related to security events and incidents. They use this information to monitor for suspicious activities, investigate security breaches, and ensure compliance with security policies.
• Compliance officers: These users are responsible for ensuring that the organization adheres to regulatory requirements. They have access to audit logs necessary for compliance reporting and auditing.
• IT support staff: These users may have limited access to audit logs relevant to troubleshooting and resolving technical issues. Their access is typically restricted to prevent unauthorized viewing of sensitive information.
• Regular users: Generally, regular users do not have access to audit logs. Their actions are logged, but regular users cannot view the logs themselves.

Restrictions based on roles are essential to maintain the integrity and confidentiality of audit logs. Only authorized personnel should have access to sensitive audit information, ensuring that the data is protected from unauthorized access and tampering.

Details included in audit logs

Audit logs comprise the following information:

• Timestamp, location, and TCP/IP protocol data
• Event description and tags
• Actors, groups, users, entities, and device identification
• Action types
• Predefined metrics
• Data access, login attempts, failures, and authentication information
• Error details
• Actions, account changes, system-wide changes, and information state changes
• Transaction details

(Understand the difference between logs & metrics.)

Use cases for audit logs: how to connect the dots

Audit logging can have four key domain applications:

• Security
• Compliance
• Accountability
• Cyber forensics

Use case: Security

In terms of cybersecurity, audit logs help to identify anomalous behavior and network traffic patterns. InfoSec teams can integrate the audit logging mechanism into their monitoring and observability solutions to extract insights on potential security incidents.

Authentication and detection of unauthorized network changes, can be achieved by testing network change actions against predefined security policies — looking at the delta. These policies define how network and IT resources are allowed to be accessed – in terms of entity, location, roles, and attributes, as well as action frequency and location.

Use case: Compliance with regulations

If your organization has to comply with external regulations, your organization may be required to keep specific audit logs and establish monitoring capabilities that test the systems for compliance by analyzing audit logs in real time. For instance:

• ISO 27001 imposes requirements for audit logging and monitoring.
• SOC1 imposes requirements for incident detection, configuration, management, and event log collection.

(See how Splunk supports organizational compliance.)

Use case: Accountability & authentication

As with standard audit procedures, audit logging is frequently used for accountability and verification of factual information. Common applications include:

• Organizational policy enforcement
• Accounting and finance
• HR policies

In this context, audit logging is an important part of analyzing how users act and the accuracy of information recorded by the systems. For example, audit logging can quickly enable systems and uncover insights into the use of financial resources across all departments. Imagine a world where all this was straightforward:

• Authorizing and spending finances.
• Understanding which users are responsible for the most spending.
• Comparing against budget allocations.

Use case: Cyber forensics

Cyber forensics is another key application domain of audit logging practices that requires the reconstruction of events and insights into a technology process. Often, this might stand up as legal evidence in a court of law.

Typically, businesses aren’t conducting cyber forensics for all their activities. Instead, we usually require cyber forensics in two situations:

• An external requirement for investigation in the form of a court subpoena
• An internal request by business executives and technical teams, perhaps around a major cyber incident or significant, unplanned downtime in a website or system

Audit logs outline the action sequences that connect a user to an action. Investigators can analyze audit logs to gain deeper insights into various scenarios and outcomes represented by the audit logs. This requires a thorough analysis of raw logging data before it is converted into insightful knowledge.

Audit logging best practices

Considering the vast volume of network, hardware, and application logs generated at scale, IT teams can be easily overwhelmed by the audit trail data. To gain the right insights with your audit log metrics data, you can adopt the following best practices:

Store all structures at a scale

Establish a data platform that can integrate and store data of all structural formats at scale. Data platform technologies such as a data lake commonly capture real-time log data streams with a schema-on-read consumption model.

Third-party analytics and monitoring tools integrate to make sense of this information in real-time while processing only the most relevant portions of audit logs data based on the tooling specifications for data structure.

Use statistical models, not predefined thresholds

Use statistical models to generalize system behavior instead of using predefined and fixed thresholds to capture data. Since the network behavior evolves continuously, models based on machine learning can continuously learn and adapt.

These models are helpful for accurate analysis of audit logs, where thresholds for anomalous behavior can be a moving target.

Secure data with eye to CIA triad

Store audit logging data in secure environments with high standards of confidentiality, integrity, and availability — known as the CIA triad. Modified audit logs and misconfigured networking systems can generate misleading information, and likely lead your log analysis to incorrect conclusions.

Infinite data storage is not sustainable

It is important to understand that data stores that integrate large volumes of real-time log data streams can grow exponentially. When designing the data platform for audit log analysis, evaluate the cost, security, and performance of your data platform against your security and compliance requirements.

Additionally, implementing quotas and limits on logging uses is crucial to managing storage efficiently. Setting quotas ensures that logging does not consume excessive resources and helps maintain system performance. Define limits based on the importance and relevance of the logs, ensuring that only critical data is retained long-term.

(And remember: you don’t need this data forever and ever — it’s not sustainable.)