PSCU Safeguards Reliability, Security Through the Splunk Platform

To improve product and service availability for more than 1,500 credit unions, PSCU needed to reduce MTTA and MTTR while boosting call-team accountability.

With the Data-to-Everything Platform, PSCU slashed MTTA and MTTR while empowering staff with mobile monitoring access to deliver support from anywhere.

PSCU is the nation’s premier payments credit union service organization. 

Using digital technology to its advantage, PSCU supports the success of more than 1,500 credit unions, representing more than 3.8 billion transactions annually. To better enable its credit unions to compete with banks, PSCU aimed to improve key IT performance metrics.

Data-Driven Outcomes
<2 min
MTTA with Splunk and VictorOps, reduced from four hours previously
3.8B+
transactions enabled by improved collaboration and enterprise functionality
1,500+
credit unions supported through mobile monitoring and real- time alerting

The PSCU Advantage

As member-owned, not-for-profit financial cooperatives, credit unions exist to serve their communities. They compete with banks by offering attractive services and rates.

Here’s where PSCU comes in. Most credit unions do not have the resources to build and host their own products, so PSCU does it for them. PSCU delivers white-label applications for online bill pay, online lending, debit and credit card programs and other financial services.

“It is critical that our services and products are available for our credit union owners,” says Earl Diem, PSCU IT operations manager.

Challenge: Improve MTTA/MTTR

PSCU saw the value in reducing MTTA — an acknowledgment that in effect says “I’m on it” when an alert is received. MTTA is a key metric for reducing downtime because it triggers incident response — which lowers mean time to repair (MTTR).

“Our people were doing the ‘rotating chair’ methodology of support, using several point tools to monitor five or six disparate systems. We recognized the need for a better alternative to give us the MTTA we sought,” Diem recalls. “We wanted to aggregate system-based alerts and gain additional traceability to more effectively manage staff accountability.”

VictorOps Slashes MTTA

PSCU solved its MTTA, MTTR and accountability challenges with VictorOps, which empowers on-call teams to find and fix problems faster with automated and insightful incident management routing, collaboration and reviews. PSCU employs VictorOps as a standard solution across 110 enterprise users. Diem keeps a graph on his wall of plummeting MTTA since PSCU started using VictorOps more than three years ago.

“In 12 months with VictorOps, our mean time to acknowledge came down from four hours to 20 minutes. Now we’re three years in, and we’re under two minutes,” Diem says. “Each PSCU IT department maintains an on-call schedule. VictorOps brought all the managers together with one tool. We understand what we’re doing, and we all use the same escalation schedule. It drives accountability.”

Bring Data to DevOps
Better manage complex, cloud-native environments and deliver exceptional user experiences armed with technical agility, speed and deep insights

Staff members use VictorOps mobility features to perform their support jobs from anywhere. “You can interact with the system from your desktop, from a laptop, from an iPad, through your phone,” Diem says. “The alerts in VictorOps give you the supporting data from the system [that] alerted. You know what went wrong even before you look at the system.”

PSCU started using VictorOps for its production environment but has extended it also to Quality Assurance and DevOps. The organization employs offshore developers in the Asia-Pacific region and India. It cannot allow system issues to interfere with productivity. Now, PSCU detects performance degradations before they turn into failures.

No matter what you do, you’re going to have failures out there. The sooner you know, the sooner you can repair it, and the better you protect your user experience.
Earl Diem
IT Operations Manager, PSCU

Extending History of Success With Splunk

PSCU has long been a customer of the Splunk platform, starting with security monitoring and Payment Card Industry (PCI) compliance — a must for financial services. PSCU’s security team uses Splunk technology to aggregate and index logs from tools monitoring network and security devices. Now, PSCU has also decided to push its operational logs into the Data-to-Everything Platform.

Because PSCU already relied on Splunk for PCI monitoring, “It didn’t make financial sense to maintain a separate tool for operations when Splunk can serve the whole enterprise,” Diem says.

Splunk’s machine data analytics, combined with incident response from VictorOps, creates a “Platform of Engagement” that helps DevOps teams innovate faster for better customer experiences. Outstanding vendor support is another advantage, as PSCU’s relationship with Splunk brings an active user community and educational resources. “I’m pretty excited about VictorOps being a part of Splunk,” Diem says.

PSCU is expanding its reliance on the Splunk platform with new use cases. One issue has been delays in detecting errors in new software releases — a problem solved by VictorOps and the Splunk App for Infrastructure.

“The errors we’re not currently seeing will bubble up, alerting into VictorOps as warnings, and we’ll have a team investigate,” Diem says. “The next natural progression after that would be Splunk IT Service Intelligence for predictive insight.”

The combination of Splunk and VictorOps software gives PSCU a powerful means to fulfill its mission of satisfying customers. “No matter what you do, you’re going to have failures out there,” Diem says. “The sooner you know, the sooner you can repair it, and the better you protect your user experience."

In 12 months with VictorOps, our mean time to acknowledge came down from four hours to 20 minutes. Now we’re three years in, and we’re under two minutes."
— Earl Diem, IT Operations Manager, PSCU
Industry: Financial Services
Do More With Splunk