Logs Are for Campfires: This Is Your Data!

The term "log" can unintentionally devalue the power that the data within the log brings to the forefront. When we reflect on the explosion of "big data," there is a lot of mystification revolving around Big Data. When we demystify big data, it is simply larger, more complex data sets from new data sources used to address problems you wouldn’t previously have the visibility to address. Applying this concept to your data in Splunk would theoretically place what is often referred to as "just logs" in the big data category. This same log data provides a treasure trove of valuable information, capturing every interaction, event, and anomaly within a system. If Big Data allows you to leverage complex data sets from new sources and address unforeseen problems, and log data helps understand system performance, identify security breaches, and optimize operational efficiency, what separates the two terms other than perception?

Click here or on the image above for an ARI Guided Demo.

Big Data Analysts, Observability Analysts, and SOC Analysts – More Similar Than You Realize

We started this article with a brief discussion of references to log data as “just logs.” This is critical because truly understanding the value of log data is crucial to the overall health of your organization—from Big Data and SOC to Observability perspectives. Now that we've addressed that, let’s further emphasize this by briefly looking at the job descriptions of analysts from all three segments. The similarities are glaring, and it always starts with not knowing what you are looking for when you start looking. Huh, Wait, What? I will expand on this later, but for now, let’s go through the job descriptions below.

Who is a Big Data Analyst?

Studies, analyzes, and reports on large amounts of data an organization has saved and preserved.

Who is a SOC Analyst?

Investigate alerts, analyze suspicious activities, and conduct in-depth analyses using various tools.

Who is an Observability Analyst?

Collects, analyzes, and interprets data to understand how a system is performing.

When you look at the job descriptions, they sound very similar. In fact, it can be argued that the three job descriptions are interchangeable. Ultimately, the requirement for each role is to study/investigate, and analyze data to help influence a business outcome. This data holds great value both historically and in the present and can be instrumental in making business decisions that impact the future direction of your organization. So, as we’ve just demonstrated, you should now see that big data and log data provide equal value and are, in fact, one and the same.

Data Sources

Let's have a look at the types of data that Splunk can ingest to provide you with valuable insights that would help you determine a business outcome for either of the roles we discussed previously. Although we will discuss most data sources, each environment is different and may have data sources that weren’t discussed here. But there is no need to worry; Splunk can ingest and normalize any data source.

Common Log Data Types

Log data is a digital record of events occurring within a system, application, or on a network device or endpoint.

• Application Logs: Provides insights into system and user activity. For example, web application logs can show which users are accessing resources, when, and what type of activity they are performing.

• System Logs: Records operating system events, such as system changes, startup messages, errors, warnings, and unexpected shutdowns. They offer insights into the operational state and efficiency of the system.

• Security Logs: Used to detect and track malicious activities throughout your environment and prevent data exfiltration and other malicious activities.

• Network Logs:

  • Firewall Logs: Records all firewall activity and security policy violations. Keep a record of allowed and denied connections, intrusion detection alerts, and more.
  • Router, Switches, and Proxy Logs: Keeps a record of all IP information within the network. Keeps track of source and destination information. For proxies, it keeps track of information in http/https requests. Methods etc.

• Audit Logs: Tracks all user actions and system state changes to ensure compliance and accountability. It provides a chronological fingerprint of all activities, which is critical for audits and compliance checks.

• Database Logs: Databases generate log data to track transactions, changes, and performance metrics. It can also provide data specific to a business function.

• Endpoint Detection and Response Logs: Endpoints provide critical forensic data, including process actions, file access information, network events, and endpoint configuration changes.

Telemetry Data Types

Telemetry is a system for collecting and analyzing data to gain insights into a system's performance. There are several types of telemetry data, including:

• Log telemetry: Logs are an essential piece of data that produces a thorough list of the events that take place within an application.
• Network telemetry: Data sources include flow logs, routing tables, application latency, and performance testing data, packet captures, or flow logs.
• Security telemetry: Data is collected from IT infrastructure sources to monitor suspicious activities, vulnerabilities, or potential breaches.
• Application telemetry: Data is collected on application performance, such as how fast the system responds, how often errors occur, and how much CPU, memory, and disk space it uses.
• Endpoint telemetry: Data sources include specific measurements such as temperature or logs of the network traffic observed at endpoint nodes.
• User telemetry: Data revolves around data related to user behaviors and interactions within systems and applications.
• Metrics: A type of telemetry data that tracks raw measurements, such as CPU usage, current Kafka lag, and percentiles of HTTP responses.

As you can see, there will always be some overlap between telemetry data, log data, and other sources of data. Although they all serve different functions, they ultimately work towards the same outcome. That outcome could be keeping your systems running, keeping your network secure, or prognosticating a future outcome based on historical data. As you can see, both log data and telemetry are mission-critical to the productivity and sustainability of your business operations. Log data is widely used in many industries and can be critical in different business verticals such as healthcare, weather forecasting, agriculture, and various research industries. An example of the importance of log data and telemetry can be medical patient metrics, such as blood pressure and heart rate data collection.

Now that we have discussed the similarities between log data and telemetry data, it should now be clear that there is no such thing as “just logs.” This is your data, which is critical to your organization's success. From the previous paragraphs, you should also be able to understand the similarities and differences in the data sources for log data and telemetry data.

In the next blog post, we’ll discuss how Splunk can add value to your existing asset data with Splunk’s Asset and Risk Intelligence.