Nexa Technologies Case Study

	Splunk helps me take what used to be 20-30 minute jobs to just a minute searching everything from one place. - Cameron Byers, System Administrator
Application Area: Business Intelligence

Customer Profile

Founded in 1999, Nexa Technologies provides a complete line of online brokerage and trading applications deployable in any market, any language, and any currency. Nexa’s product line includes direct access trading platforms, browser-basted trading solutions, back-office and account management systems, real-time and historical Market data, FIX execution services, and more. To learn about Nexa, please visit www.nexatech.com.

Nonstop trade support

Mark Cales and Cameron Byers administer the back-end trading applications that Nexa develops and hosts on behalf of their brokerage customers. Their primary role is to keep the applications and servers running. But they spend a large portion of each day answering business questions escalated from a 15-person brokerage support team stationed in California, Toronto and Texas. Did the customer’s bid for 200 shares of GOOG at 11am go through or not? Did it execute at the right price? Or, why was he told he couldn’t make the trade? Too often, Mark and Cameron need to look through application logs to find the answer.

Business Challenge

Before Splunk, answering these business questions was a cumbersome process that involved downloading data from sensitive production systems. Mark and Cameron refused all but the most critical requests, yet they still handled ten to fifteen every day. Each request took at least ten minutes; some took an hour or more. That’s a couple of hours lost each day by a highly skilled two-person team.

Don’t give no for an answer

Mark and Cameron (and their management) knew they needed to say "yes" more often, so that broker customers would get better service and Nexa’s correct execution of its trades would be more evident. But they also needed a more efficient way to search application logs so each request would take less of their time away from their IT tasks.

What if there was a solution, they wondered, that would let Nexa's brokerage support team handle these requests themselves, leaving IT out of the picture?

Technical Requirements

System Architecture

Nexa has fifty Windows Server hosts running custom trading software in a highly secure data center in Irvine, CA. This application generates a single log. This log contains some events in the industry-standard FIX (Financial Industry Exchange) log format that are the official record of the exchange settlement. The log also includes other events specific to Nexa’s application. These events provide more details about each order and about the application itself. But if you're not an IT guy, they can be hard to read:

It's easy to see why the trade support staff escalated questions about transaction details.

Before Splunk

Each Nexa application instance wrote its log to a local file on one of the fifty Windows hosts. The IT team had also written homegrown scripts that let a privileged user—Mark or Cameron—log into the production environment and retrieve a file containing all log events for a particular timerange that matched some simple criteria. Mark or Cameron would then open this file on their desktop and use vim, a command line text editor, to locate the relevant log records to answer the customer’s question.

Running the script to retrieve the file for each request took about 5 minutes. Searching through it manually to answer the question added at least 5 minutes more. Sometimes the data retrieved by the first pass at production wouldn’t be enough, or the trade hadn’t happened within the time window retrieved. This could easily lead to an hour-long hunt.

Splunk at Nexa Technologies

Bill Warshaw and Ville Maanpaa, a manager and developer working with Mark and Cameron at Nexa, found Splunk at the end of 2005, just after Splunk Professional 1.0 was first released. Ville downloaded the 30-day trial, installed it on a Linux desktop and experimented with some of their logs. He found that Splunk indexed Nexa's custom logs easily and enabled him to search everything via a simple, Web-based interface. Ville and Bill envisioned their brokerage support team using a few saved searches in this interface to answer customer questions themselves.

Bill and Ville contacted Splunk support to help them figure out how to get live data from their production Windows servers. They were also curious to see if there was a way to extend Splunk’s back-end processing to insert the real names of exchanges, such as "Toronto," in place of cryptic codes like "K" that appear in the FIX format. That would make the logs much more usable for the trade support staff.

Splunk support helped them craft a workable solution, so Nexa purchased Splunk Professional in December. But since they planned to change their logging mechanism, they postponed their rollout to the release of Splunk 2.0 in early 2006.

To get data into Splunk, Ville modified Nexa's application to use Kiwi, a 3rd party Windows logging utility. Kiwi provides a .DLL libary that applications can call to record a log event. Kiwi runs locally on each of Nexa's Windows servers, and forwards events in syslog UDP format to a central Linux server that runs Splunk.

Cameron handled the Splunk host setup. He turned on Splunk's syslogUDP input module, which listens on UDP port 514 directly during the guided installation. That was about it for customizations. The entire Splunk 2.0 installation took less than 10 minutes.

Mark and Cameron have now moved all of their day-to-day searches to Splunk. That instantly saves 5 minutes per request in setup time. It also avoids return trips to retrieve more logs when the initial set doesn’t answer the question.

Next Steps

This summer, they'll be training the support team in a series of remote sessions on how to Splunk for transaction records. They expect to get out of the business of validating trades so they can focus on IT issues—for which they'll also use Splunk.

Scenarios

Did this trade execute? At the right price?

A broker wants to know if a trade he placed actually happened. The Nexa support person goes to the Saved Splunks menu and chooses a canned search that finds all FIX settlement events. He adds the specific order ID to the search. Splunk returns a definitive answer: There either is or isn't an event that matches, even for settlements that were completed only a few seconds earlier. It's fast enough that it can be done while the broker is on the phone.

If the customer doesn't have their order ID, the Nexa rep can search the transaction logs for a customer ID and look for recent events that do have an order ID in them. He can click on the order ID and add the Saved Splunk to perform the search for a settlement event.

If a customer wants all details for their order, the rep can search by order ID and export the results to a text file to email to the customer.

Why did the Nexa system make the decision it did about this broker's current buy power?

The Nexa system implements business rules that consider many criteria about a broker’s current account and trading to decide whether that broker has sufficient buy power to perform a specific buy. Brokers often call to find out what the buy power calculation was.

The Nexa rep will handle this request similarly to the transaction validation above. The only difference is the rep will look for a series of log messages preceding the trade activity for the order ID in question. These will show three buy power calculation messages. Splunk’s View Source link, which shows a selected event in the surrounding stream of events with which it was loaded, lets them quickly spot the calculation messages prior to a trade.