By Splunk May 18, 2017
Setting alert logic is often an extensive process of trial and error. It can be difficult to know when alerts are triggered in production or how to determine when attention is required when multiple alerts are firing. Even with a monitoring solution flexible to support more complicated alert logic, the dynamic environments of cloud infrastructure, containers, orchestration platforms, and open-source middleware can complicate alerting. Are you confident that the threshold or conditions you are using will fire an alert when you need it most?
SignalFx’s new Alert Preview compresses the iterative trial-and-error cycles in setting alert logic. With Alert Preview, you can instantly confirm when alerts are triggered using the selected threshold or conditions. Immediately tune your alerts to make sure they fire when they should — and not when they shouldn’t.
Get to the Right Alert
A typical process for building the right alert includes setting up the threshold or conditions, getting notified when it is triggered (or worse, not getting notified when it should have triggered), adjusting the trigger condition if necessary, and waiting for the next alert. Alert Preview accelerates this trial and error process.
After creating or editing the logic that should trigger an alert, run the conditions against recent data from your environment and see where alerts would have fired or cleared. Determine whether the settings are appropriate for your requirements or use the preview to fine-tune to your specifications. Gain more visibility into your data and get a closer look at exactly what conditions trigger the alert.
Having a preview of your alert logic means you gain more confidence that your alerts will fire only when they should. Alert Preview lets you see the frequency of alerting based on live data from your environment. Eliminate the cost and confusion of unnecessarily firing noisy alerts or false positives.
Fine-Tune to Your Environment
Alert Preview lets you create, deploy, and tune alerts so you don’t have to wait until an incident actually occurs to see whether the alert fires. While we know there are operations and development teams who have years of experience with their specific service patterns and cloud environments, unexpected issues occasionally arise. These teams need to quickly modify thresholds or conditions of an existing alert based on past experience.
Unexpected issues surface the reality that some alerts are either too sensitive (they trigger alerts too often) or not sensitive enough (they don’t trigger alert when you want to be notified of an issue). In either case, use Alert Preview to more accurately modify alert settings so it triggers the appropriate number of alerts. Adjust settings until you see a preview that makes sense for your data.
Cloud operations teams can also easily leverage more powerful ways to alert than the standard practice of comparing against static thresholds based on the behaviors of the environment. Once a metric is identified, Alert Preview makes it easy to set an alert condition by choosing from a menu of Built-in Alert Conditions based on patterns or trends of the data. Read more about new Built-in Alert Conditions here.
Day-to-day operations of any cloud environment requires creating alerts that fire when they should and not when they shouldn’t. How confident are you that the alert logic will notify before an outage affect your customers, business, or reputation? Alert storms can cripple a team’s ability to quickly and effectively remediate issues. In the midst of a real outage, can you afford for your team to be too fatigued to respond?
A preview into when alerts are triggered and cleared based on recent data from your environment is the first step towards gaining confidence that your alerts are relevant and accurate. It also enables you to tune thresholds and conditions to your requirements and removes the fatigue of dealing with alert storms and false positives. Alert Preview in SignalFx dramatically reduces the cycle time to create, deploy, and tune alerts for your cloud application.
Thanks,
Jessica Feng