The Machine Data Journey: How Cisco Data Fabric Helps Teams Land, Discover, Promote, and Search Data

Platform Michelle Corpora , Courtney Wright

Machine data keeps growing, leaving teams with difficult choices: discard data to manage costs, bury it in cold storage where it’s hard to find, or pay a premium to make every dataset immediately searchable. When an investigation or AI initiative raises a new question, the context they need may be out of reach.

That’s the challenge Cisco Data Fabric powered by the Splunk Platform is designed to address. It’s an architecture for federating machine data so teams can analyze it where it lives, connect context across domains, and put operational data to work in analytics and AI.

To maximize the value of your data without breaking your budget, data must move through a deliberate, logical journey

Overview of the federated search options for the Splunk platform

Cisco Data Fabric acts as the connective tissue for this entire journey, eliminating the traditional tradeoffs between cost, retention, and accessibility. Here’s how its underlying capabilities work together to help teams get more value from their data:

Prepare and Route Data for Its Purpose

Data Management helps teams collect, process, govern, and route operational data. Teams can filter noisy data, mask sensitive fields, transform telemetry, and direct data based on its value and intended use. This gives teams more control over what they send downstream and how data is handled. It can help reduce unnecessary processing and storage, strengthen governance, and prepare data for its intended use.

Our recent autonomous data pipelines innovations help teams get the right data to the right place for the right purpose, with less manual toil and more confidence in the quality of data.

A Splunk admin at Atea, a European IT infrastructure company, said, “Auto-Schematization showed strong promise for simplifying data onboarding. The workflow was intuitive, and the ability to identify logs, map them to the right data model, and generate props/transforms could significantly reduce manual effort for teams that manage complex security data.”

AI does not replace the administrator. The workflow gives administrators a faster starting point, clearer recommendations, and reviewable outputs they can inspect and edit before using.

fabric-1.png

Land, Discover and Promote Machine Data With Room To Adapt

Machine Data Lake provides a governed storage environment designed for machine data. Teams can retain full-fidelity data without making an upfront commitment to index everything. When a workload calls for faster, repeated searches or structured analysis, they can promote the relevant data to a higher-performance tier, as needed.

For example, a security team can keep older telemetry in Machine Data Lake for occasional investigations, such as a compliance audit. If that same data becomes central to a recurring dashboard or a longer-running security investigation that needs faster, repeated searches, the team can promote the relevant data to a higher-performance tier, such as a Splunk index. This lets teams make retention and performance decisions around the data they actually need to use. A dataset does not have to be treated as either “index everything now” or “keep it where no one can use it.”

fabric-2.png

In addition, the Catalog can complement this approach by providing context across the data estate, helping teams find and understand the data they need. Metadata helps teams understand what data is available and discover relevant datasets for investigations. For a broader look at how Machine Data Lake and Catalog help teams retain, discover, and activate machine data, explore this article.

fabric-3.png

Search Across Supported Data Locations

Federated Search extends the Splunk analytics experience across third-party and select Splunk-managed storage environments. Teams can query data where it resides without first moving everything into Splunk-managed storage. This can be especially useful when an investigation or audit needs context from data that lives in a different location or is retained in a lower-cost tier. Teams can bring relevant data into the analysis while avoiding unnecessary data duplication and movement.

fabric-4.jpg

With Federated Search for Amazon S3 in Splunk Cloud Platform, Autodesk now sends high-value, frequently searched log data to Splunk for real-time analysis, while less critical, infrequently accessed logs — such as ones that are only used once to troubleshoot a single issue — are stored in Amazon S3 data lakes, saving the company roughly 28% in overall ingest costs.

By migrating just one of our existing search tools to Splunk Federated Search for S3, we saved 78%. - Jyoti Kumar, Principal Engineer at Autodesk.

Consider a security team investigating a service issue. To land incoming telemetry, Data Management prepares and routes it, while Machine Data Lake can provide a destination for retaining full-fidelity data in raw, open format to discover relevant datasets, the team can use Catalog to see what data is available across Splunk indexes, Machine Data Lake, and supported third-party environments.

If older telemetry becomes important to a recurring dashboard or a repeated security investigation, the team can promote the relevant data from Machine Data Lake raw tier to a higher-performance tier, such as a Splunk index or Analytics Table. To search data already stored in a supported external location, the team can use Federated Search to query it in place. Promoted data can be searched through Splunk analytics.

Each capability answers a different question:

Start With the Question You Need To Answer

Begin with one high-value security, IT, network, or AI use case. Define the decision your team needs to support, then consider how current the data must be, how often it will be searched, how long it needs to be retained, and where it lives today. Those requirements can guide how data is prepared, where it is retained, and how you can access it for analysis. Get started quickly with your data journey.

Explore the Splunk platform. Happy Splunking!

Related Articles

Don’t boil the ocean: A technologist’s take on prioritisation in sustainability
Security
5 Minute Read

Don’t boil the ocean: A technologist’s take on prioritisation in sustainability

Even if manufacturing isn’t close to your heart, you’d have to be pretty cold not to care about sustainability in 2023. Let's get a technologist’s take on prioritisation in sustainability.
Matching AI Strengths to Blue Team Needs
Security
6 Minute Read

Matching AI Strengths to Blue Team Needs

Discover how AI and Large Language Models (LLMs) enhance cybersecurity operations for Blue Teams.
Baselining and Beyond: What's New in OT Security Add-On v2.2
Security
5 Minute Read

Baselining and Beyond: What's New in OT Security Add-On v2.2

Splunk has released a new version of OT Security Add-On, designed to help organizations understand their cybersecurity risks, improve their security monitoring, and better detect and react to industrial cybersecurity threats.