Tokyo Stock Exchange Delivers World-Class Performance With Data-Driven Analytics

Executive Summary

A Japanese icon and one of the world’s leading financial markets, Tokyo Stock Exchange (TSE) has been driving fair securities trading since 1878. Accompanying the launch of its next-generation cash equity trading system “arrowhead” in 2010, TSE strives to sustain world-class reliability by enhancing performance analysis and server capacity monitoring. Since deploying the Splunk® platform, TSE has:

• Boosted overall performance with data analytics that deliver details to the millisecond
• Improved in-house data manageability with analysis that’s tens of times faster
• Increased cost-effectiveness and achieved measurable return

Advanced Data Analytics to the Rescue

Committed to fostering economic and social prosperity through technological innovation, TSE launched its arrowhead cash equity trading system to let everyone trade stocks in a fair, secure way. This system raised expectations for system reliability, which demanded better monitoring of server capacity. Yet data analysis at TSE was a long, manual process that sometime took days to complete, requiring administrators to download data from arrowhead and write custom scripts in C or Perl. Due to the sheer volume of data, TSE also needed more sophisticated statistical techniques to get accurate results.

To facilitate data analysis and increase visibility into its business, TSE chose Splunk technology, which has improved operational efficiency, agility and flexibility across the organization.

Performance Analysis With Millisecond Precision

The Splunk platform provides TSE full visibility of its machine data. Splunk’s forwarder unites massive amounts of business transaction data and resource-related logs (like CPU and memory usage logs) from the arrowhead servers for automatic analysis, enabling the TSE team to process 50 million orders every day.

The Splunk dashboard displays analysis results through a web-based graphical interface, allowing TSE to easily monitor resources, transaction queuing, backup and operational status — such as number of orders processed per second, number of market status notifications per second, notification delays and order acceptance response time.

TSE no longer has to analyze its ever-rising volumes of data by hand. Providing greater visibility and accuracy than TSE’s prior processes, the Splunk platform visualizes changes in the number of orders processed and resource usage down to the millisecond. This allows TSE to get a complete picture of system health, resolving problems before they arise.

With Splunk, TSE has solved its manageability challenges at speed and scale. “We need a platform that is fully manageable in-house, flexible enough to handle different situations and able to parse data in minutes or even seconds,” says Atsushi Yamamoto, head of infrastructure and operations in the IT development department at TSE.

Splunk technology makes these goals possible by automatically storing data to the indexer, allowing TSE to quickly retrieve trade transaction information with Splunk’s search processing language (SPL). “We are running as many as 80,000 transactions per second, but Splunk lets us see all data in real time,” explains Yamamoto. “Since arrowhead processes every order in about 200 microseconds, even one second of waiting time is too long, but Splunk keeps pace with our needs by generating results tens of times faster than our old system.”

From saving time to increasing data availability, TSE is also reaping other benefits across the organization. “SPL comes with a series of functions for calculating percentile values. Its summary indexing also lets us extract values in advance and quickly get information,” says Yamamoto. “We now streamline coordination with peripheral systems through the intuitive applications programming interface while automating repetitive manual tasks. By compiling a simple search query, the information will immediately appear in Excel.”

TSE has found the Splunk platform to be highly cost effective. “Splunk gathers and interprets diverse sets of data at amazing speed with just four servers, and it’s super scalable.” Yamamoto says. “Our legacy system needed a middleware for processing high-speed data streams, but Splunk Enterprise does the same job on its own.”

Splunk technology also helps measure the values of arrowhead. With the click of a button, Yamamoto views all statistical information in charts and evaluates the effectiveness of arrowhead at a single glance. These easy-to-use data visualizations minimize the need for Excel and other software, saving time and simplifying collaboration for Yamamoto’s team. “The Splunk community is also a valuable resource for us,” he adds. “By having access to an extensive collection of reference materials and a huge user base, we get the answer to any question in no time.”

Moving forward, Yamamoto will use Splunk plug-ins to collaborate with middleware and business intelligence tools while incorporating other data, such as network device logs and operational data, into the Splunk platform. With Splunk at the core of its business analytics, TSE will continue to unlock new opportunities for growth and data-driven success.