false
Engineering

Principal Cloud Lab Engineer - Observability

  • - Remote
The Observability Lab Engineer will be responsible for creating and maintaining Splunk IT observability solutions in the Splunk Center of Excellence (COE) lab. This role involves installation and configuration of Splunk tools that accurately reflect real-world conditions to support testing, development, and analysis efforts. The engineer will primarily collaborate with the Splunk technical marketing team to install, configure and maintain solutions in the COE lab for use creating demo content and “day in the live” situations to help share the power and value of Splunk with long time users, potential customers, industry analysts, and partners.

Key Responsibilities

  • Design, build, and maintain dynamic lab environments across AWS, GCP, and Azure to support observability, APM, and security testing.
  • Work closely with technical marketing, product engineering, and QA to define infrastructure requirements that reflect real customer deployments.
  • Automate provisioning and teardown of cloud lab resources using Terraform, Ansible, or cloud-native tools (e.g., CloudFormation, Deployment Manager).
  • Implement observability and monitoring pipelines (e.g., Splunk, OpenTelemetry, Prometheus) to simulate production-level visibility into test environments.
  • Create “day in the life” operational scenarios to emulate common IT issues, incidents, or performance degradation.
  • Ensure high availability, cost efficiency, and version control of cloud environments via scripting and automation.
  • Maintain documentation for lab environment setup, automation scripts, and testbed configurations.
  • Provide support during live demos, webinars, and product showcases with reliable, demo-ready infrastructure.

Minimum Qualifications

  • Bachelor’s degree in Computer Science, Information Systems, or equivalent professional experience.
  • 5+ years of experience managing cloud-based lab/test environments, preferably in hybrid or multi-cloud settings.
  • Proficiency in cloud platforms (AWS, GCP, Azure) and IaC tools (Terraform, CloudFormation, Ansible).
  • Strong experience with observability stacks, especially Splunk, plus exposure to APM, log aggregation, and metrics systems.
  • Scripting experience in Python, Bash, or PowerShell to automate deployment, monitoring, and validation.
  • Experience with containerized environments (Kubernetes, Docker) and service mesh configurations.
  • Solid troubleshooting skills and knowledge of networking, security groups, and CI/CD pipelines.
  • Strong documentation and communication skills for cross-functional collaboration.
  • Comfortable supporting live demos and resolving environment-related blockers under pressure.
  • Experience with performance benchmarking tools and load generators (e.g., JMeter, Locust)
  • Familiarity with demo orchestration frameworks or event injection tools for synthetic data
  • Knowledge of GitOps practices and DevOps pipelines for continuous environment refresh
  • Splunk Core Certified User or Power User (or similar tool experience) - nice to have

 

Splunk, a Cisco company, is an Equal Opportunity Employer and all qualified applicants will receive consideration for employment without regard to race, color, religion, gender, sexual orientation, national origin, genetic information, age, disability, veteran status, or any other legally protected basis.  

Splunk's Hiring Practices

Splunk turns machine data into answers. Organizations use market-leading Splunk solutions with machine learning to solve their toughest IT, Internet of Things and security challenges.

We value diversity, equity, and inclusion at Splunk and are committed to equal employment opportunity. Qualified applicants receive consideration for employment without regard to race, religion, color, national origin, ancestry, sex, gender, gender identity, gender expression, sexual orientation, marital status, age, physical or mental disability or medical condition, genetic information, veteran status, or any other consideration made unlawful by federal, state or local laws. We consider qualified applicants with criminal histories, consistent with legal requirements. Click here to review the US Department of Labor’s EEO is The Law notice. Please click here to review Splunk’s Affirmative Action Policy Statement. If you need assistance or an accommodation to apply or during the hiring process, please let us know by completing our Accommodation Request form.

Splunk also has policies in place to protect the personal information candidates disclose to us as part of the application process. Please click here to review Splunk’s Career Site Privacy Policy.

Splunk does not discriminate against employees or applicants because they have inquired about, discussed, or disclosed their own pay or the pay of another employee or applicant. Please click here to review Splunk’s Pay Transparency Nondiscrimination Provision.

Splunk is committed to the health and safety of our employees and customers. Splunk is impacted by the mandates outlined for U.S. Government contractors in President Biden’s Path out of the Pandemic: COVID-19 Action Plan. As a result, Splunk requires U.S. employees, whether assigned to an office or 100% remote, to provide proof of full vaccination, as defined by the CDC. Splunk provides reasonable accommodations for employees who have qualifying medical or religious reasons.

Splunk is also committed to providing access to all individuals who are seeking information from our website. Any individual using assistive technology (such as a screen reader, Braille reader, etc.) who experiences difficulty accessing information on any part of Splunk’s website should send comments to accessiblecareers@splunk.com. Please include the nature of the accessibility problem and your e-mail or contact address. If the accessibility problem involves a particular page, the message should include the URL of that page.

Splunk doesn't accept unsolicited agency resumes and won't pay fees to any third-party agency or firm that doesn't have a signed agreement with Splunk.

To check on your application click here.

DIVE DEEPER

Find out what makes Splunk such a great place to work

box1 box1
Our Values

Splunkers are encouraged and empowered to be Innovative, passionate, disruptive, open and fun.

Learn More
box2 box2
Benefits and Wellbeing

Our benefits are designed to support your physical, financial, emotional and mental wellbeing.

Explore Splunk Benefits
box3 box3
Early Talent Program

Intern with people you want to hang out with, even outside the office.

Learn More
box3 box3

Our Blog

Hear from Splunkers on the latest.

Read the Blog
box2 box2
Diversity, Equity, Inclusion & Belonging

Learn about Splunk’s commitment to creating a culture of belonging.

See Our Approach
box1 box1
LinkedIn

Follow Splunk on LinkedIn for job announcements, company news and more.

Follow Us