What Is Federated Analytics?

Don’t move data to the analytics — instead, we move analytics to the data. This is Federated Analytics.

Let’s discuss what federated analytics means and how it can help change your data approach in context of the data challenges facing your organization.

(See how Splunk Federated Analytics makes it easy to search and correlate data across Splunk and third party data lakes.)

What is federated analytics?

Federated Analytics is the concept and practice of performing data analysis tasks on decentralized devices, connected data sources, or cloud points locally — instead of moving that data to a central location for analysis.

The federated analytics approach preserves privacy. That’s because the analytics models process raw information on the device itself. It does not require active collaboration between the devices — user data is not transmitted between devices or a centralized third-party analytics server.

How federated analytics works

Federated analytics processes local data to update the parameters of its local analytics model. The model updates across all devices are aggregated and then shared to a backend system that updates the parameters of a global analytics model.

This process is iterative: the global model updates guide the next cycle of model parameter updates for the local device models.

At all stages of the federated analytics process, privacy is preserved for the raw user information — that’s possible because the data itself never leaves the device. The model parameter information extracted from each device is also insufficient to reverse engineer sensitive user information. You can make it especially secure if this information is encrypted at the device level prior to transmission.

When we speak of federated analytics as a generalized version of distributed intelligence and a collaborative computing paradigm, it includes non-ML models used for:

• Edge analytics
• IoT
• Low-power devices

These models are lightweight and require low compute power to analyze data in real-time. These models suffice for analyzing trends and extrapolation from historical data that may not be very high-dimensional (where each data observation describes a high number of features or variables).

For example, IoT sensors measuring individual characteristics at a network node, such as:

• Temperature
• Data rate

Federated analytics vs. federated learning

Federated Analytics is different from the core distributed intelligence paradigm of Federated Learning, which follows the same principles for:

• Model updates
• Privacy-preserving data processing

However, the difference is that the models are typically large neural networks. The global model combining all parameter updates typically exceeds millions of parameters, or billions in the case of Large Language Models.

(Related reading: federated data, federated search & federated AI.)

In this keynote, Gary Steele, EVP, General Manager, Splunk, discusses the role of federated analytics in modernized SOCs at RSA Conference, 2024.

Challenges with data today

Consider the following key challenges:

• Data privacy regulations. Organizations are forced to keep data in certain geographies (aka data residency), each enforcing a different set of privacy regulations for its users and their data. Many organizations are not prepared to handle these limitations on their own, especially when the SOC requires data transfers to enable threat detection and cybersecurity use cases.
• Data gravity. Transferring large files and data streams to a central location in real-time is expensive — in terms of bandwidth consumption, storage redundancy, and versioning. Worse? It exposes security risks during data transmission.
• Hybrid multi-cloud complexity. Data architectures are complex and the modern IT environment is fragmented. Many organizations operate a mix of private, public, hybrid and multi-cloud environments. Critical data resides on-prem, where legacy applications may have weak authentication systems as well as limited integration to actually support scalable and efficient centralized analytics.
• Point solutions. Organizations operate a variety of distributed threat detection point solutions spread across the network, but it is difficult to make sense of the collective security posture and threat landscape in real-time.
• Converging data. SOC uses several tools for different security functions. These tools are siloed and yet organizations are expected to collaborate efficiently in DevOps environments.

Solving data challenges

To solve these challenges, we need to change our approach. That means that any model where you centralize all your data — whether for IT operations or the SOC — the model of centralizing all data is fundamentally gone.

Data remains critical, of course — instead of moving data to the centralized analytics platform, let’s move analytics to the data.

For example, in the context of a SOC, a modern one will be built on the following three pillars:

• A single platform for threat detection, investigation and response. Fewer tools that are tightly integrated in an environment that is open and extensible.
• Automation and AI that filters the noise for SOC managers and security analysts to investigate the most impactful threats. Analysts focus on rapid action, using automation and AI as key tools to elevate the role of SOC experts as defenders.
• Federation that allows analytics performed on data wherever it lives. Data can be stored in low-cost storage systems. Alerts and patterns are extracted through an AI data preprocessing pipeline designed uniquely for every data source type. The SOC can choose where the data must reside based on organizational, policy and regulatory constraints — as opposed to the technical limitations of an analytics platform that requires a centralized database or data lake system.

Indeed, it is precisely the state of technology available today that allowsus to change our approach in many systems and drive analytics to the edge.

When to use federated analytics

Now, let's review some of the key downstream applications of Federated Analytics. The demands for federated analytics come from a few primary drivers.

Data is distributed

For an analytics model to impute data from a true data distribution, it must have access to either:

• The raw data itself
• Information that very closely resembles the true data distribution

Access to this information is limited in privacy-sensitive use cases, such as serving ads based on recent financial transactions. This information is also widely distributed, across devices, client endpoints such as browsers and apps — each with their own layers of separation for privacy and security.

A federated analytics service can access distributed data sources and perform the necessary computing operations locally within all distributed devices and client endpoints, which allows for real-time analytics processing.

Only the model parameter updates are communicated to a backend server. It is this piece that allows for the efficient execution of decision controls between a centralized analytics service running a global model at the back end and the front-end user device.

This means that federated analytics and AI models can be trained on domain-specific and contextual problems that require continual learning.

Privacy awareness and regulations

As people become more aware of privacy issues, they are opting out of features and services that provide unrestricted data to third-party services. These services access limited information from sources such as:

• Cookies
• Data aggregation partnerships
• Browser fingerprinting
• API requests for approved data collection sources

Privacy regulations also restrict organizations from sharing sensitive end-user data with their partners. Any loss of user information or a data breach incident can lead to hefty fines, as well as loss of brand loyalty and consumer interest.

This is where federated analytics can fill the gap: data never leaves a user’s device — the model parameters are updated based on true user information to guide analytics services. The service provider and their partners never access user information outside of the user-controlled device.

That means federated analytics could help narrow the scope of your regulatory compliance.

Analytics models are large

Of course ML-based analytics models are large, but did you know that even the non-ML analytics models can be large in several ways?

The feature space and the number of variables captured by device sensors locally can be large and complex.

• Captured in real time, sensor information can grow exponentially as the number of variables are increased.
• For traditional non-ML models that rely on rules-based decision trees, the model size also grows exponentially with every branch (rule) added to the model.

User information and sensor data is stored in hierarchical structures with underlying dependencies and relationships, which must be captured and analyzed separately for every model deployment. For a large collective user base, the global models are large enough to warrant dedicated computing resources in a cloud-based data center.

This is where federated analytics presents itself as a mechanism for performance optimization: It reduces the need (and cost, and resources) for large-scale data transfer. That’s because the analytics processing takes place on smaller datasets locally to update a relatively small set of model parameters for faster real-time insights.

In this model, compliance and privacy controls are strictly enforced at the user end. 

Federated analytics can challenge quality & highlight bias

The key challenge for federated analytics relates to data quality. The training data is heavily skewed with a bias toward the device user. This is different from the conventional approach to training or updating an analytics model, which involves the training process to run on the entire data distribution collectively.

The challenge for engineering teams is to understand the patterns of bias and then regulating model updates in ways that would compensate for the model bias.