Defense and Intelligence Agencies

The Power of Big Data for Defense and Intelligence Agencies

Government defense and intelligence agencies are tasked with collecting, analyzing and storing massive amounts of data to detect and correlate patterns of activities related to security threats. These agencies must also utilize systems that can handle extremely granular role-based access controls (RBAC) so that only those that ‘need-to-know’ have access to the right data at the right time.

Splunk Enterprise is the industry standard product for big data analysis is widely deployed across hundreds of government agencies to help pass their FISMA assessments and:

  • Detect patterns and anomalies across terabytes of raw data in real time without specialized skills, up front data normalizations or fixed schemas
  • Can use Hunk for Hadoop and NoSQL data stores to provide a unified view or your data
  • Provide automated monitoring for NIST 800-53 controls supporting the 800-37 risk management framework
  • Supports continuous monitoring and the acquisition of context data from any event from any layer of the IT structure


Traditional perimeter-based defense approaches are ill-equipped to handle today's sophisticated security threats originating from inside an agency. When agencies replace their security information and event management (SIEM) with Splunk it is out of a larger need to become more proactive about security events.

Splunk's big data platform:

  • Captures security and operations log data from mission-critical custom applications
  • Helps incident response and forensics teams get to root cause analysis faster in the face of higher data volumes and more data types
  • Detects patterns and discovers malicious behavior and attacks not seen by signature and rule-based systems
  • Provides value and fast incident response by combining data from traditional security point solutions and credentialed user-to-machine interactions to see events in context.

Insider Threats

The need to detect insider threats has forced agencies to look for new ways to understand complex user behavior. Knowing the difference between willful acts and innocent mistakes requires understanding when user activity is abnormal in the broader context of employee behavior.

Identifying activities that are unauthorized or suspicious is done through the use of Splunk's statistical analysis commands on very large data volumes.

With Splunk you can:

  • Access any data via web services or direct database access
  • Index tens of terabytes of data per day and apply statistical analysis to baseline data to watch for outlier behaviors.
  • Perform damage assessments by understanding traffic and communication patterns in network log data, data from emails and other file transfer methods
  • Access customs and border patrol data to reveal foreign travel
  • Load data into Splunk from databases that contain financial disclosure data obtained through public facing credit services

Continuous Evaluation

Individuals who have access to sensitive government information should be monitored for activities that could be construed as malicious behaviors in the context of other external information. Unauthorized travel, wild fluctuations in credit scores, major relationship changes and starting a business are just a few of the activities that can be tracked in a variety of IT systems.

Splunk can monitor and analyze this kind of IT data by watching for anomalous behaviors and performing on-demand correlations to other external data sources both inside and outside the agency. This approach can help you distinguish between an accidental policy violation and someone with malicious intent.

Internet of Things and the Data-Driven Battlefield

In 2015, the Army will start testing TALOS (Tactical Assault Light Operator Suit) in the field for deployment in 2018. Sensor data from the suit can provide information on the operating status of suit hydraulics and batteries. Next steps often discussed include monitoring soldier vital signs and hydration.

This data can be correlated with GPS data, weapon performance data and soldier health information to provide location and condition information for any unit. Units can be monitored in near real-time and proactively resupplied as data from RFID tagged equipment is added to the mix. RFID data can be used to track inventory to support supply chain management and notify suppliers to restock. Look-ups to manufacturer data can reveal performance information to understand which lot number from which manufacturer may be underperforming.