Monitoring Telecommunications Network with Splunk

Introduction

For telecommunications service providers, the ability to gain granular insights into network behavior has become a fundamental requirement for organisations striving for operational excellence and resilience. These organisations need the tools to effectively monitor and analyse network performance, reliability, and security in near real-time. Telco networks are complex, multi-layered systems, encompassing various domains such as the radio access network (RAN), the transport network, the core network, and the service layer. Each domain layer incorporates equipment from multiple OEMs like Cisco, Ericsson, Huawei, and Nokia etc and thus presents unique monitoring challenges due to diverse technologies, vendor-specific implementations, and the sheer volume of data generated.  Effectively monitoring these layers requires a holistic approach that can correlate data from disparate sources to provide a unified view of network performance and health.

Splunk can help address these challenges by providing a platform that can ingest and process data from these diverse sources. By enabling a unified cross domain view of the network, Splunk allows telcos to:

• Monitor complex multi-vendor network environments comprising equipment from multiple vendors with inherent capability to handle disparate data formats and communication protocols.
• Empower network administrators and security teams to derive meaningful intelligence from the continuous flow of network telemetry dataSplunk's robust suite of search, reporting, alerting, and dashboarding functionalities further 
• Create a unified approach to monitoring heterogeneous network infrastructures by centralising the analysis of network data.

This blog aims to provide a comprehensive guide on how Splunk can be employed to effectively collect network telemetry data from telecommunication network devices. The subsequent sections will delve into various data collection methodologies, the network protocols it supports, the availability of out-of-the-box connectors or relevant applications, strategies for managing the high data volumes characteristic of telco environments, and pertinent case studies illustrating Splunk's application in this area.

Splunk's Core Mechanisms for Network Telemetry Ingestion

Splunk provides several established methods for collecting network data, each offering distinct advantages depending on the specific data required and the capabilities of the network devices. These methods cater to various data sources and architectural preferences within telecommunication environments.

The Universal Forwarder is a lightweight agent that can be installed directly on network devices or on intermediary servers. Its primary function is to collect data from diverse sources, such as system logs and application logs, as well as performance metrics, and securely forward this data to Splunk indexers for processing and analysis. The Universal Forwarder is compatible with a wide range of operating systems, including Linux and Windows, which are commonly found within telecommunication infrastructures.

The HTTP Event Collector (HEC) offers a secure and efficient method for sending application events and logs directly to Splunk over HTTP or HTTPS protocols. HEC utilises a token-based authentication model, ensuring secure data transmission. This method is particularly well-suited for custom applications that may be running on network equipment or within management systems. Notably, HEC is designed to handle high volumes of events, making it appropriate for the potentially large data streams encountered in telco networks.

Splunk Connect for Syslog (SC4S) presents a modern, containerised approach to syslog data ingestion. SC4S is essentially a pre-configured syslog server (based on syslog-ng) encapsulated within a container. This framework simplifies the process of collecting syslog data from a multitude of network devices by providing a standardised ingestion pipeline.  

Splunk Connect for SNMP (SC4SNMP) provides a similarly modern, containerised solution for gathering SNMP data from network devices. Deployed at the network edge, SC4SNMP is designed for high availability and seamless integration with Splunk Enterprise, Splunk Cloud, and Splunk Infrastructure Monitoring. A key advantage of SC4SNMP is its ability to provide context-rich information about the interfaces and performance metrics of network devices without requiring users to manually construct SNMP queries.

The OpenTelemetry Collector offers a technology-agnostic approach to receiving, processing, and exporting telemetry data, encompassing logs, metrics, and traces. The Collector operates based on a pipeline concept, with receivers to gather data, processors to manipulate it, and exporters to send it to backend systems like Splunk. The Splunk Distribution of the OpenTelemetry Collector extends the open-source project by including components tailored for specific vendors and platforms, facilitating data collection from diverse environments. A significant benefit of OpenTelemetry is its ability to standardise observability data formats, thereby minimising vendor lock-in.

Finally, Splunk Stream is an application designed to capture network packet data, including valuable protocols like NetFlow and IPFIX. Through deep packet inspection, Splunk Stream can extract a wealth of protocol attributes from the captured network traffic. This capability makes it highly relevant for real-time traffic analysis, security monitoring, and gaining granular insights into network communications.

The availability of these diverse ingestion methods underscores Splunk's flexibility in accommodating the wide array of data sources and protocols prevalent in telecommunication networks. Organisations can select the most appropriate method based on the specific type of data, the capabilities of their network devices, and their unique monitoring objectives. Furthermore, the increasing prominence of containerised solutions like SC4S and SC4SNMP reflects a broader trend towards modern, scalable, and more easily managed data collection architectures, which are particularly advantageous in the large and dynamic environments typical of telecommunications.

Here's a breakdown of commonly used protocols and collection methods for network data:

The following table summarises the Splunk network telemetry collection methods by OEM:

Technical Approaches for Network Telemetry Data Collection

Here's a more detailed look at methods to get data into Splunk, with some additional suggestions:

• Install a Universal Forwarder: Deploy a Splunk Universal Forwarder on the data-generating device (e.g., network equipment, server). The forwarder collects data locally and sends it to a Splunk indexer. This is a highly scalable and reliable method, especially for large volumes of data. Forwarders can also perform some basic pre-processing.
• Forward Syslog events: Configure network devices to send Syslog data to a Syslog collector (which could be a dedicated server or Splunk itself). Splunk can ingest Syslog data directly or via a collector like rsyslog or Splunk Connect for Syslog (SC4S), which provides enhanced Syslog handling capabilities.
• Use Vendor APIs: Many network devices and systems provide APIs (often RESTful) for accessing data. You can use scripts or Splunk's HTTP Event Collector (HEC) to retrieve data from these APIs and ingest it into Splunk. This method is useful for obtaining specific, structured data.
• Database Integration: For devices that store data in databases, use Splunk DB Connect to extract data and index it in Splunk. This allows you to bring in data from relational databases.
• HTTP Event Collector (HEC): Send data directly to Splunk using HTTP or HTTPS requests. HEC is efficient and scalable, and it's particularly well-suited for sending data from applications and cloud services.
• Streaming Telemetry: Increasingly, network devices support streaming telemetry, which pushes data in real-time (e.g., using gRPC). Splunk can ingest this high-velocity data for very granular monitoring.
• Message Queues: Integrate with message queueing systems (e.g., Kafka, RabbitMQ) to consume data that's being published by network devices or other systems. This provides a decoupled and scalable way to ingest data. 

Recommendations and Best Practices for onboarding data into Splunk

To effectively set up and optimise Splunk for collecting network telemetry data, several recommendations and best practices should be followed:

• Prioritise Standard Protocols: Begin by leveraging Syslog and SNMP as these protocols are widely supported by most network devices for reporting operational events and device status.
• Utilise Splunk Connectors: Employ Splunk Connect for Syslog (SC4S) and Splunk Connect for SNMP (SC4SNMP) to simplify and enhance the scalability of data collection from network devices.
• Explore Splunkbase Thoroughly: Conduct comprehensive searches on Splunkbase for any vendor-specific apps or Technology Add-ons (TAs) that might streamline the integration process and provide pre-built dashboards and reports tailored for OEM devices in your network.
• Consult Vendor Documentation: Refer to the official documentation provided by your OEMs for specific guidance on configuring Syslog, SNMP, and NetFlow/IPFIX on their respective devices to ensure proper data export to Splunk.
• Normalise Data with CIM: Map the collected telemetry data to the Splunk Common Information Model (CIM). This normalisation enables consistent analysis and correlation of data across different sources and vendors within the Splunk platform.
• Implement Effective Filtering: Utilise Heavy Forwarders or other data enrichment tools to filter out any unnecessary or redundant data before it reaches the Splunk indexers. This practice helps to reduce the overall data volume and optimise indexing performance.
• Optimise Data Retention Policies: Define clear and appropriate data retention policies based on regulatory requirements, compliance standards, and specific business needs to effectively manage storage costs associated with the potentially high data volumes in telco environments.
• Secure Data Transmission: Ensure that sensitive telemetry data transmitted to Splunk is secured using appropriate protocols such as TLS (Transport Layer Security) to maintain confidentiality and integrity.
• Monitor Splunk Performance Regularly: Continuously monitor the health and performance of the Splunk infrastructure itself to ensure that it can effectively handle the data ingestion and analysis demands of the telecommunication network.
• Engage with the Splunk Community: Actively participate in the Splunk Community forums and leverage the available resources to gain insights into best practices and solutions specific to network telemetry in telecommunication environments.

Case Studies of Splunk in Telecommunication Networks

Splunk helps global telcos manage their complex network operations effectively, these case studies highlight the platform's versatility and the tangible benefits it provides to telco organisations.

Telenor, a major telecommunications service provider, has successfully deployed Splunk to enhance incident investigation, streamline troubleshooting processes, and bolster its security posture. The implementation of Splunk has led to benefits such as quicker and easier resolution of business-critical issues, enhanced security capabilities, and increased overall service availability. Telenor's network operations team utilises Splunk dashboards to visualise network health and proactively monitors for error events and unusual patterns. 

CenturyLink (now Lumen Technologies) has also adopted Splunk for mission-critical monitoring, achieving improved executive-level visibility into their IT and business operations and significantly reducing incident resolution times. Splunk has enabled CenturyLink to centralise monitoring across a complex environment involving both in-house and third-party applications. The platform provides real-time insights into the performance and errors within their billing system, which has streamlined operations for their DevOps personnel. Moreover, Splunk has empowered their call centers with real-time dashboards, allowing them to independently assess their performance and identify procedural issues, reducing their reliance on IT for initial problem assessment.

These case studies collectively demonstrate that Splunk is a well-established and proven solution for a wide range of applications within the telecommunications industry. The successes reported by companies like Telenor and CenturyLink underscore the practical advantages of using Splunk to manage the complexities of modern telecom infrastructures, leading to tangible improvements in incident response, service availability, and overall operational efficiency. This makes Splunk a strategic asset for achieving comprehensive observability and driving data-driven decision-making in the telecommunications landscape.

To learn more about Splunk for communications service providers visit our website or contact your Splunk representative.

Regards,

Gaurav Gupta

Industry Strategist (Telco & Retail)