Hyperledger Fabric Security Monitoring with Splunk

In this post, we demonstrate how to set up effective security monitoring of your Hyperledger Fabric infrastructure. We identify some common threats, recognize key data sources to monitor, and walk through using Splunk to ingest and visualize your data. This post follows Introducing Splunk App for Hyperledger Fabric and highlights the use of the app for security monitoring of blockchain infrastructure.  We will address smart contract/chaincode security & monitoring in a follow-up post. 

Threats

Blockchain infrastructure incorporates a diverse set of technologies, across a distributed network, which means it faces some unique challenges. Some of the most significant infrastructure threats include Denial of Service (DoS), Key Theft, Network Partitioning, Consensus Manipulation, and Blockchain Integrity Attacks. We show indicators and data sources required to detect DoS, Consensus Manipulation and Ledger Manipulation attacks and demonstrate results from an emulated DoS attack.

Key Data Sources

In addition to the data sources that are already important to monitor in any enterprise environment, blockchain systems produce huge amounts of additional data that should be monitored. 

Generally this data is separated into on-chain and off-chain data. On-chain data consists of the ledger data, which is persistent and resistant to tampering from network-wide replication. Off-chain data is often ephemeral and may not be widely distributed. Examples of off-chain data include the state database, network traffic, as well as node metrics and logs. The table below shows how a combination of data from the ledger, logs, and metrics can be used to indicate different threats.

Getting Started with Hyperledger Fabric Monitoring

We can easily analyze Hyperledger Fabric’s ledger, log, and metric data with the following tools. Splunk Connect for Hyperledger Fabric ingests ledger and metric data from a Hyperledger Fabric deployment. The Splunk Docker logging driver can be used to send container logs to Splunk. Finally, Splunk App for Hyperledger Fabric facilitates the analysis of this data in Splunk. For a walkthrough of the main features of Splunk App for Hyperledger Fabric read Introducing Splunk App for Hyperledger Fabric.

Once you have the app running, click on the Security Monitoring dashboard to be presented with a high level view of several threat indicators. Of note, indicators of DoS include trends in transaction latency & throughput, unique senders, as well as open gRPC connections. Orphaned blocks may indicate attacks on Blockchain Integrity. Finally, indicators of Consensus Manipulation include configuration updates and consensus leader changes.
 

^{A view of the Security Monitoring dashboard}

Of course this dashboard is only a sample of what is possible. You can further expand on the searches in any of these dashboards yourself, using events captured from node logs or ledger data, along with Prometheus/StatsD Hyperledger Fabric metrics.

Example: Detecting DoS Attacks

Now, we’ll demonstrate how our monitored indicators respond to an emulated DoS attack. In this scenario, an authorized user has their keys compromised and begins spamming the network with transactions.

We will be paying particular attention to the transaction latency, throughput, and number of open connections.  First, we’ll look at the normal case where a single client is sending 10 transactions per second.
 

^{Normal Case: Transaction and Connection Metrics}

Next, we’ll have a single client open up persistent 1000 connections each performing 1 query per minute. Here, we see that the transaction latency starts to increase, transactions per second decreases, and the number of open connections increases. At this point, it may be difficult to determine if this is reflective of a high period of load, misconfiguration, or a denial of service attack.
 

^{Adversary Case: Transaction and Connection Metrics}

We can investigate further in the Infrastructure Health and Monitoring Dashboard, where we see connection and I/O timeout errors.
 

^{Infrastructure Health and Monitoring: I/O Timeouts}

Because we noticed a large number of open connections, we should query Splunk to see the distribution of gRPC message subjects and addresses. When we perform this search we see a large discrepancy in message count — indicating that “User1@buttercup.example.com,L=San Francisco,ST=California,C=US” is likely compromised or misconfigured and should be investigated further.

^{gRPC Message Counts}

Conclusion

Comprehensive monitoring can help to secure your blockchain infrastructure by quickly identifying threats and facilitating investigations. Monitoring is only one part of a secure software development life cycle, and it does not replace the need for independent audits or penetration tests. We showed how you can use the Splunk App for Hyperledger Fabric to monitor the security of your Hyperledger Fabric deployments and facilitate incident investigation and remediation.

Stay tuned for a deep dive in contract/chaincode security & monitoring in a follow-up post. If you need assistance with Hyperledger analytics connect with us at blockchain@splunk.com.

----------------------------------------------------
Thanks!
Chris Cordi