Splunk at Pegasus Solutions

New Operational Visibility Drives the Travel and Leisure Industry

Overview

Industry

  • Travel and Hospitality - IT Services

Splunk Use Cases

  • Application Management—troubleshooting, performance monitoring, code optimization
  • Operations Management—help desk dashboard

Business Impact

  • Reduced inquiry-tracking time from nearly 40 hours a week to minutes
  • Delivered real-time insight to operations team with critical telemetry dashboard for tracking system health and performance
  • Provided developers a secure, single point of access to application data and error logs enabling measurement of how various configurations affect network and application performance
  • Enabled self-service views into transaction data to empower help desk staff

Data Sources

  • Syslog from hosts via a central forwarder
  • Syslog from network devices
  • Scripted inputs of business logic in raw XML
  • Over 20 J2EE-based custom application logs

Splunk Configuration

  • 2 Sun x45 Thunder Humpers
  • Forwarders on 50 hosts
  • 50 commodity Intel boxes
  • Multiple Sun T-2000s and M5000s

Download

Success Story
Splunk at Pegasus

“It used to take hours, even days to track transactions for some customers. Using Splunk, our support teams can typically respond to issues immediately with realtime data and insight.”
Peter Ehlke
Principal Systems Engineer, Pegasus Solutions

The Business

Pegasus Solutions is a company that you likely often encounter without ever knowing it. As the world's leading IT services provider to the travel and hospitality industry, Pegasus delivers reservations, commission and payment processing, and hotel marketing services worldwide to nearly 100,000 hotels, 1,000 travel websites and a majority of the world's travel agencies.

Pegasus empowers these organizations to accomplish the once-impossible feat of correlating room rates, inventories, payments and commissions for reservations systems, agents and hospitality chains. As a critical component in the travel and hospitality revenue engine, Pegasus demands peak performance and ultimate reliability to maintain customer satisfaction and service levels. Providing excellent service to industry icons such as Marriott, Fairmont and Orbitz is a tall order, and to do it Pegasus requires the very best in IT monitoring and measurement.

Challenges

Transaction tracing, especially with four to five billion per month, was no small feat for Pegasus. Due to sheer volume, they could only manage 24 hours of transactions online and keep three days of back-up transactions on-premise. However, verifying a guest activity such as a cancellation or confirmation is a common occurrence.

Any request for tracking a transaction older than three days required Pegasus to make an off-premise request to deliver the tape. Once on-site, Pegasus was faced with reviewing logs across 15 different systems to reconstruct and locate the request.

Other "discomforts" Pegasus experienced included access to log data directly on production systems, force-fitting homegrown logging formats into database-driven tools, and running customer reports in batches.

Enter Splunk

Embracing the need for modernization, Pegasus pioneered the first major redesign of the central reservation system (CRS) in more than twenty years. Its new RezView(r) NG was recently named one of the Top 100 IT projects by InfoWorld magazine, and its innovative deployment relied largely on Splunk.

Application Troubleshooting

Developers securely access error logs via Splunk, so they no longer log into the live environment. Splunk accounts are mapped to Active Directory, so administrators only have to manage permissions in one place. User data is masked at index time to mitigate security risk and ensure user data remains private. Administrators also use Splunk to measure how various configurations affect network and application performance.

Operations Management

Saved searches and reports help Customer Support resolve problems quickly, and the operations team address issues proactively.

Pegasus developed a "canary-in-the-coalmine" dashboard that includes all the critical metrics and telemetry to assess system performance at a glance. This comprehensive dashboard indicates the overall system state through a traffic metric—the number of transactions per second. The operations team gauges the performance impact on users measuring server response times. If response times seem high, they know if it's a network problem because the same script also monitors google.com.

At the next level, the dashboard shows an average response time aggregated across all transaction types as well as the standard deviation, so they know if the whole system is sluggish or if it's only a few outlier transactions. Pegasus get a further breakdown through more scripted inputs that gather and display the execution time for each transaction type. Scripting that reaches deeper into the system for the number of transactions in the message queue provides early warning. If this number starts to rise, Pegasus knows a problem is brewing.

Breakthroughs

Insights

Splunk has delivered new insight to Pegasus and its customers. With Splunk, the support team can typically respond to customers while they're still on the phone. A year's worth of transactions available online reduces inquiry-tracking time from nearly 40 hours a week to minutes—freeing up one full-time employee and dramatically improving customer service.

Proactive customer service

Splunk also offers comfort for the future. Rather than relying on scheduled reports, Pegasus plans to offer portlets where customers can view their data in real-time. So, if customers want to measure the impact of a promotion, they can see results immediately—and change offers dynamically based on real-time analysis of results.