Case Study

University of North Carolina at Chapel Hill Gains Deep Insight Into Mission-Critical Applications

Executive Summary

The University of North Carolina at Chapel Hill (UNC-CH) is the nation’s first public university and a global higher education leader. The 739-acre university serves more than 29,000 students and employs nearly 12,000 faculty and staff—all of whom depend heavily on the reliability, security, access and performance of the campus network. UNC-CH Information Technology Services (ITS) needed a solution to better manage, troubleshoot and secure its dynamic IT infrastructure. Since deploying Splunk Enterprise, UNC-CH has seen benefits including: 

  • Simplified views of complex system interdependencies
  • Distributed access to central repository of machine data
  • Savings of hundreds of hours in troubleshooting
Challenges
    • Provide operational visibility into critical applications for IT, departmental and executive management
    • Reduce the time required for troubleshooting and remediation
    • Provide wider access to log data without risking server integrity
    • Deliver means to monitor all systems and devices in campus network
    • Provide self-service troubleshooting and risk-free access to log data
Business Impact
    • Saves hundreds of hours per year by proactively alerting and reducing troubleshooting time
    • Improves operational efficiency and system availability
    • Helps achieve & enhance educational mission
    • Provides access to data from all campus systems in near real time
    • Reduces troubleshooting time
    • Provides risk-free access to production data for developers and other power users
Data Sources
    • Firewalls and IPS/IDS systems
    • Microsoft Active Directory, Exchange Server
    • Oracle Weblogic, Tuxedo, PeopleSoft Campus Solutions
    • RedHat OpenShift PaaS
    • SAS Business Intelligence
    • Java applications
    • Sakai Learning Management System

Why Splunk

An extensive wired and wireless campus network supports UNC-CH’s more than 40,000 users and over 100,000 connected devices in more than 300 buildings. The University network also supports many mission-critical applications, including PeopleSoft Campus Solutions, Microsoft Exchange and Microsoft Active Directory. Splunk Enterprise first found a home within the ITS organization in 2009 as part of a log centralization project. Since then, the University’s Splunk license has grown from an initial 50GB per day to 900GB. Adoption of the Splunk platform has expanded to more than 400 regular users in multiple university departments including ITS, the College of Arts and Sciences, School of Medicine and the School of Public Health. “Before Splunk we had anecdotal discussions and finger pointing because no one really knew what was going on with their systems,” recalls Patrick Casey, manager of Middleware Services within the ITS Infrastructure and Operations group at UNC-CH. “With Splunk Enterprise, we have a factual-based reporting of events—we now know exactly what took place. Splunk software enables us to take something that is technically complex and make it understandable and useful through dashboards and reports so that anyone can use it to make meaningful decisions.”

Splunk value drives widespread adoption

The Splunk platform provides the University with a single view into operations without admins having to log into individual hosts, allows for faster investigation of potential breaches, security event correlations and easier identity management. Splunk software also enables admins to pinpoint firewall problems and provides easier access to production logs for developers. The middleware team is promoting widespread Splunk adoption by providing a portal that delivers access to shared tools and apps that can be used directly or easily adapted to meet specific needs.

“We’re trying to encourage as many teams as possible to use the Splunk platform because it helps improve our efficiency in so many areas,” notes Dave Safian, senior solutions engineer in Middleware Services and a Certified Splunk Administrator. “For instance, and in conjunction with our Active Directory team, we’ve set up an account lockout tool within Splunk. Before, when someone went to the help desk for lockout assistance, the help desk would need to submit it to Tier 3 support to troubleshoot, which is very expensive and time consuming. Now, the help desk can use the Splunk tool we built to correlate lockout information in order to pinpoint problems. It not only saves time and money, but we’ve improved customer satisfaction because users get an answer right away instead of hours later.” 

Dashboards deliver business insights at a glance

The middleware team developed a Splunk dashboard system that aggregates KPIs for the University’s critical Sakai Learning Management System, providing visual information about the health of servers running Sakai applications. The information is used to adjust performance parameters to improve the learning experience. 

PeopleSoft Campus Solutions powers the ConnectCarolina application and is used to deliver important services for students, staff and faculty. Splunk forwarders are installed on every PeopleSoft server and device, resulting in more than 10,000 unique log files each week. Splunk dashboards are used to monitor and troubleshoot the CarolinaConnect system, providing near real-time information that accelerates troubleshooting by delivering critical information on system status and hotspots that need attention.

“Our customer base is extremely tech savvy and very active on all forms of social media,” concludes Casey. “When there’s a problem with our systems and we’re not fulfilling our mission to educate students, that hurts our reputation and adversely impacts the educational experience. We are using Splunk Enterprise to identify and solve some of our biggest operational challenges and make critical operational decisions. From our admins to our CIO, we rely on the Splunk platform to make huge volumes of data meaningful to our decision makers at every level.”