Puget Sound Energy Improves Response Time With Real-Time Data, Saves With Facility Intelligence

Think for a moment: How vital are heat and electricity in your home?

For more than 145 years, PSE has kept homes warm and lights on across a 6,000-square-mile service area, supplying electricity and gas to approximately 2 million customers throughout western Washington. The company needed to improve response during power outages, monitor and gain full visibility into its facility data, and consolidate tools in its network operations center (NOC).

From Power Outages to Data Centers

Chris Perez, enterprise technology solutions advisor at PSE, is part of the hybrid facilities and IT team. “Our team handles change management for our data centers, as well as the power distribution, cable, plant, and overall facilities maintenance of 150 communications sites and rooms in 70 office locations,” Perez says.

“Our customers’ uptime is the most critical part of what we do. Power outages don’t just affect residential customers, they affect hospitals and EMS services. Having visibility into that is key,” Perez says. However, legacy software and multiple tools spanning environments didn’t allow necessary correlations, making it challenging to find the root cause when power outages hit. PSE also needed a better monitoring solution for its critical data centers, and a more reliable way to ensure resource availability in its conference rooms.

PSE’s adoption of Splunk has been organic and continues to grow. A small team with a limited implementation started monitoring customer-facing websites. The team recognized the potential for tackling additional challenges and championed broader adoption across the organization.

Improved Response During Power Outages With Real-Time Data

When power outages occur, PSE needs to quickly understand the affected service areas and the number of customers without power. Previously, the System Operations team depended on legacy software with limited visibility. To solve this, PSE worked with third-party vendor Arcus Data to leverage splunk for Industrial IoT and deploy a holistic outage management system (OMS).

Now, with Splunk dashboards providing actionable, real-time information, the team can proactively respond and stay updated about power restoration, reducing call center volume and improving customer satisfaction. Splunk has also helped improve PSE’s emergency response during outages — previous dashboards weren’t reliable and the data wasn’t always relevant. Splunk dashboards now provide consistent visibility into many data sources that have improved the team’s ability to prioritize when dispatching crews. The OMS use case was such a success that PSE is evaluating Splunk for monitoring and aggregating additional data from critical OMS applications to achieve end-to-end health analysis.

Data about temperature, fan speeds, HVAC and environment readings generated by PSE’s data centers was previously siloed in different systems from multiple vendors. Limited data access and no view across data centers meant there was no way to see key performance indicators or identify issues within individual facilities. To remedy this, PSE aggregated all general alerts with log data, gaining full visibility into the performance of its data centers.

The team also needed to make sure employees could always collaborate regardless of location. This required monitoring the health of hundreds of conference rooms, each with up to 60 pieces of equipment, such as occupancy sensors, room schedulers and video switchers. Before Splunk, PSE lacked real-time data about equipment problems and occupancy, relying on user feedback and logs from individual devices. There was also no way to see how often equipment was being used, which resulted in unnecessary maintenance costs. With Splunk, the team can access real-time conference room data at any time and fix issues faster.

Splunk also helps keep costs down with just-in-time maintenance — dashboards identify which equipment requires immediate attention, eliminating unnecessary site visits and saving on maintenance costs.

Before adopting Splunk, PSE’s IT NOC team used multiple tools, which required constant employee training. Standardizing on Splunk has reduced training time and costs, and increased the consistency and reliability of information gathering, sharing and response. The Splunk platform has been such a success that the team was able to sunset some legacy tools, resulting in an annual ROI of $135,000 and an anticipated savings of $500,000 through further consolidation.

PSE sees Splunk’s capabilities continuing to extend from IT into operational technology (OT). As PSE pulls data from the data center programmable logic controllers (PLCs) — specifically actuator, analog sensor and fan data — to monitor the health of the physical equipment, IT can now better communicate with local facility teams. “The most powerful thing about Splunk is its ability to bridge the gap between IT and OT — Splunk provides a frame of reference to each side and a common language that fosters collaboration,” Perez says.