By Teneil Lawrence November 08, 2023
With the rapid adoption of cloud, distributed systems and microservices are standard, resulting in increasingly complex environments. Once straightforward troubleshooting workflows have become chaotic, frustrating, and time-consuming. When something breaks, multiple teams are called to the table to prove they’re “not it”; each with their singular view of the problem. This siloed approach results in missed issues when no one has the full context you need to get to the root cause of the issue. MTTR mounts and your job satisfaction dwindles.
When Every Second Counts
Imagine your engineering team has launched a new application. You might need to visualize all the dependencies, from your infrastructure (e.g., servers) to microservices, monitor specific performance metrics by different groupings to align to business priorities for the quarter, and set up alerting and response workflows so you’re prepared to quickly respond if any of these metrics fall out of range. Then the moment you’ve been preparing for occurs - you get a storm of alerts and need to start troubleshooting to find the root cause.
Legacy monitoring and first-generation observability tools can complicate this process. The former doesn’t provide visibility to the cloud and, with an ad-hoc collection of different cloud vendor tools, getting to a shared context is nearly impossible. The latter might get you some visibility but delayed analytics, a lack of intelligent tagging, and misalignment from one dashboard to the next could mean visibility gaps, alerting delays, and costly downtime for the business. When one minute of downtime can cost a business up to $9,000, every second counts.
A Better Start, A Quicker Finish
Splunk’s Observability portfolio is built to help you overcome these cloud-induced challenges. When you need to detect and troubleshoot issues in a cloud environment quickly and confidently, only Splunk can deliver. Cloud network to code-level visibility, a real-time metrics engine, full fidelity distributed tracing, directed troubleshooting, and intelligent analytics all work together to help you shorten the time it takes to resolve issues before they negatively impact your business.
All this is easier said than done, which is why, in this series, we’re going to show you how you can use Splunk to quickly find the needle in the haystack and isolate the root cause of any problem when something breaks in your complex, cloud environment. Step-by-step, we’ll guide you through how to:
- Get started visualizing your cloud services
- Index span tags to prep for troubleshooting
- Automate performance scans so you can take action as soon as something breaks
- Investigate when a problem occurs
Speak the Language
Some key concepts, some unique to Splunk, that you’ll come across in this series include:
- Business Workflow: A set of correlated traces that track a transaction or user flow of particular interest.
- Dashboard: Dashboards are groupings of charts and visualizations of metrics.
- MetricSets: Key indicators, such as request rate, error rate, and durations, that are calculated based on traces and spans in Splunk APM.
- Navigator: A collection of resources that lets you monitor metrics and logs across various instances of your services. Resources in a navigator include a full list of entities, dashboards, related alerts and detectors, and service dependencies.
- Span Tags: Key-value pairs added to spans through instrumentation. In OpenTelemetry, these key-value pairs are known as attributes.
- Tag Spotlight: A top-down view of your services indexed by unique tags relevant to your business based on indexed span tags.
- Trace Analyzer: Search through all traces from all instrumented services to find the exact trace you’re looking for.