Key takeaways
In the fast-moving world of CI/CD pipelines, microservice architectures, and container orchestration, software changes rapidly. What exists in a codebase today might be gone next week.
At this scale and speed, it’s impossible for development teams to manually track every line of code and every new piece of functionality. That’s why Infrastructure as Code (IaC) became a foundational DevOps practice: replacing manual provisioning with version-controlled, machine-readable configuration files to manage infrastructure.
The benefits of infrastructure as code:
Rapid delivery of value
Enhanced security (lower risk)
Simplified and scalable provisioning
Optimized changes
Consistency across the organization
Better tracking of resource usage
So here’s the question: Why don’t we apply the same principles to observability?
Just like IaC applies software development practices to infrastructure management, observability as code (OaC) does the same for observability.
Instead of creating dashboards, alerts, and integrations manually through observability platform UIs, OaC defines them in human-readable configuration files. These files can be:
Version-controlled
Reviewed through pull requests
Deployed alongside your code
With the need to interact directly with UIs or CLIs now eliminated, it’s easy to maintain, update, and deploy changes across environments.
Observability as code helps teams overcome the inefficiencies and inconsistencies that sometimes result from traditional observability. It offers benefits similar to infrastructure as code:
Shareable, standardized, and consistent
Scalable, automated, and repeatable
Secure
Let’s break these down.
Code-managed observability resources can be packaged into reusable modules, allowing different services and different environments to share configurations. This means:
Observing the end result in your observability platform is more consistent, across the entire organization.
Changes made once are easily applied.
For resources like dashboards, charts, or alerts, consistency means you can easily interpret data across services, products, and teams. During incidents, the ability to find key metrics and compare them across environments means:
Faster insight into root cause
Decreased time to incident resolution
Additionally, with observability configurations version-controlled, you get:
Audit logs and change history (more context for better insight)
Updates handled through standard code merge/review processes, just like every other code change
Simple rollbacks if you need to revert changes
With observability as code, configurations live alongside the codebases they monitor. No more scrambling through hundreds of alerts in a UI to adjust thresholds. Instead, engineers can:
Edit config files directly and redeploy observability resources.
Apply changes across multiple resources and multiple environments in minutes.
All of this is especially valuable in microservice architectures, where each service exists in multiple environments (development, staging, production). OaC speeds up development and issue detection and resolution. That’s because now teams have the same insight and visibility into every environment, and potentially across every team.
OaC also tightens observability-related security in several ways:
API keys and credentials are stored securely with other code secrets.
Admin access isn't required for observability configuration.
Administrators can better track usage (and cost) of observability tools.
The most common tools for OaC are Terraform and OpenTofu.
These tools allow you to define observability resources in config files alongside your infrastructure. These files describe what you want your monitoring setup to look like — the dashboards, alerts, charts, integrations you need — and the tool creates and maintains them.
Once you initialize, preview, and apply your configuration, your observability resources are created automatically. The best part is that any future updates follow the same predictable process.
Begin with your most critical alerts or dashboards. Expand gradually. You don’t need to convert everything at once.
(See how a tiered approach to observability helps prioritize investments, reduce complexity, and focus on what matters most.)
Develop templates for common monitoring scenarios. This ensures consistency when adapted across different services and teams. It also reduces the time to onboard new services.
Observability deployment should be part of your standard CI/CD process. New code goes out with the monitoring code built-in — no code should be deployed to production without its observability resources.
Before deployment, ensure the pipeline verifies all changes are instrumented and observable.
Store all configurations in source control with your app code. This enables team-wide collaboration through reviews and merge requests.
You can even designate observability champions to build expertise and enforce standards. That enables visibility and more opportunity across the business.
Treat observability configurations like any other code. Centralize credential management and protect access tokens with the same rigor as your application logic.
In a world of microservices and distributed architectures, observability as code tackles the complexity of scale. It improves:
Resource consistency
Collaboration
Maintenance
Speed of development and incident resolution
If you’re new to Terraform, check out Terraform’s documentation and tutorials.
Are you already using Terraform? Explore our step-by-step guide to managing observability with Splunk Observability Cloud — or better yet, try it for yourself with this free 14-day trial of Splunk Observability Cloud. Begin your OaC journey today!
Say goodbye to blind spots, guesswork, and swivel-chair monitoring. With Splunk Observability Cloud and AI Assistant, correlate all your metrics, logs, and traces automatically and in one place.
Observability as Code (OaC) means defining your monitoring setup (like dashboards, alerts, and integrations) in version-controlled configuration files, so you can automate and standardize how you manage observability across all your environments.
OaC helps you save time, reduce errors, and keep things consistent by letting you automate monitoring setups, easily track changes, and quickly apply updates across different teams and environments.
You can use tools like Terraform or OpenTofu to define your observability resources as code, and many observability platforms (including Splunk) offer integrations to support this approach.
Yes, you can use Terraform with Splunk: there are specific Terraform providers that let you define, deploy, and manage Splunk observability resources directly from configuration files.
The main benefits are faster setup, better consistency, easier collaboration, improved security, and the ability to scale and update your monitoring quickly as your systems grow or change
See an error or have a suggestion? Please let us know by emailing splunkblogs@cisco.com.
This posting does not necessarily represent Splunk's position, strategies or opinion.
The world’s leading organizations rely on Splunk, a Cisco company, to continuously strengthen digital resilience with our unified security and observability platform, powered by industry-leading AI.
Our customers trust Splunk’s award-winning security and observability solutions to secure and improve the reliability of their complex digital environments, at any scale.
Get the latest articles from Splunk straight to your inbox.
© 2005 - 2025 Splunk LLC All rights reserved.
