SECURITY

Hyperledger Fabric Security Monitoring with Splunk

In this post, we demonstrate how to set up effective security monitoring of your Hyperledger Fabric infrastructure. We identify some common threats, recognize key data sources to monitor, and walk through using Splunk to ingest and visualize your data. This post follows Introducing Splunk App for Hyperledger Fabric and highlights the use of the app for security monitoring of blockchain infrastructure.  We will address smart contract/chaincode security & monitoring in a follow-up post. 

Threats

Blockchain infrastructure incorporates a diverse set of technologies, across a distributed network, which means it faces some unique challenges. Some of the most significant infrastructure threats include Denial of Service (DoS), Key Theft, Network Partitioning, Consensus Manipulation, and Blockchain Integrity Attacks. We show indicators and data sources required to detect DoS, Consensus Manipulation and Ledger Manipulation attacks and demonstrate results from an emulated DoS attack.

Key Data Sources

In addition to the data sources that are already important to monitor in any enterprise environment, blockchain systems produce huge amounts of additional data that should be monitored. 

Generally this data is separated into on-chain and off-chain data. On-chain data consists of the ledger data, which is persistent and resistant to tampering from network-wide replication. Off-chain data is often ephemeral and may not be widely distributed. Examples of off-chain data include the state database, network traffic, as well as node metrics and logs. The table below shows how a combination of data from the ledger, logs, and metrics can be used to indicate different threats.

 

Threat

Indicators

Data Source

Denial of Service

Tx throughput & latency

Block latency

# Senders

# Open connections

Block headers,

Node metrics

Consensus Manipulation

Changes in chain config

Leadership Elections

Blocks
Node logs

Ledger Manipulation

Orphaned blocks

Block headers

 

 


Getting Started with Hyperledger Fabric Monitoring

We can easily analyze Hyperledger Fabric’s ledger, log, and metric data with the following tools. Splunk Connect for Hyperledger Fabric ingests ledger and metric data from a Hyperledger Fabric deployment. The Splunk Docker logging driver can be used to send container logs to Splunk. Finally, Splunk App for Hyperledger Fabric facilitates the analysis of this data in Splunk. For a walkthrough of the main features of Splunk App for Hyperledger Fabric read Introducing Splunk App for Hyperledger Fabric.

Once you have the app running, click on the Security Monitoring dashboard to be presented with a high level view of several threat indicators. Of note, indicators of DoS include trends in transaction latency & throughput, unique senders, as well as open gRPC connections. Orphaned blocks may indicate attacks on Blockchain Integrity. Finally, indicators of Consensus Manipulation include configuration updates and consensus leader changes.
 


A view of the Security Monitoring dashboard


Of course this dashboard is only a sample of what is possible. You can further expand on the searches in any of these dashboards yourself, using events captured from node logs or ledger data, along with Prometheus/StatsD Hyperledger Fabric metrics.

Example: Detecting DoS Attacks

Now, we’ll demonstrate how our monitored indicators respond to an emulated DoS attack. In this scenario, an authorized user has their keys compromised and begins spamming the network with transactions.

We will be paying particular attention to the transaction latency, throughput, and number of open connections.  First, we’ll look at the normal case where a single client is sending 10 transactions per second.
 


Normal Case: Transaction and Connection Metrics


Next, we’ll have a single client open up persistent 1000 connections each performing 1 query per minute. Here, we see that the transaction latency starts to increase, transactions per second decreases, and the number of open connections increases. At this point, it may be difficult to determine if this is reflective of a high period of load, misconfiguration, or a denial of service attack.
 


Adversary Case: Transaction and Connection Metrics


We can investigate further in the Infrastructure Health and Monitoring Dashboard, where we see connection and I/O timeout errors.
 


Infrastructure Health and Monitoring: I/O Timeouts

Because we noticed a large number of open connections, we should query Splunk to see the distribution of gRPC message subjects and addresses. When we perform this search we see a large discrepancy in message count — indicating that “User1@buttercup.example.com,L=San Francisco,ST=California,C=US” is likely compromised or misconfigured and should be investigated further.



gRPC Message Counts

 

Conclusion

Comprehensive monitoring can help to secure your blockchain infrastructure by quickly identifying threats and facilitating investigations. Monitoring is only one part of a secure software development life cycle, and it does not replace the need for independent audits or penetration tests. We showed how you can use the Splunk App for Hyperledger Fabric to monitor the security of your Hyperledger Fabric deployments and facilitate incident investigation and remediation.

Stay tuned for a deep dive in contract/chaincode security & monitoring in a follow-up post. If you need assistance with Hyperledger analytics connect with us at blockchain@splunk.com.

Chris Cordi
Posted by

Chris Cordi

Chris Cordi is a Software Engineer on Splunk’s blockchain team. Before joining Splunk, he was a cybersecurity researcher at Sandia National Laboratories where he worked on DLT security. He currently leads research & engineering efforts for security features of Splunk’s blockchain integrations, focusing mainly on Ethereum and Hyperledger Fabric. He also serves as co-chair of the Enterprise Ethereum Alliance’s EthTrust Security Levels Working Group. He is passionate about smart contract and blockchain network security, and in his spare time enjoys running and hiking with his family. He has a B.S. and M.S. in Computer Science from the University of Illinois at Urbana-Champaign.

TAGS
Show All Tags
Show Less Tags