The person in this role will be responsible for creating and maintaining Splunk security solutions in the Splunk Center of Excellence (COE) lab. This role involves installation and configuration of Splunk tools that accurately reflect real-world conditions to support testing, development, and analysis efforts. The engineer will primarily collaborate with the Splunk technical marketing team to install, configure and maintain solutions in the COE lab for use creating demo content and “day in the live” situations to help share the power and value of Splunk with long time users, potential customers, industry analysts, and partners.
Key Responsibilities:
Architect, deploy, and manage hybrid cloud lab infrastructure using Splunk Cloud, Splunk Enterprise, and associated ecosystem tools.
Lead efforts to design lab environments that emulate enterprise scenarios, ensuring scalability, data fidelity, and realistic security/IT operations workflows.
Collaborate with technical marketing, engineering, and sales engineering teams to define reference architectures and enable compelling product demos and simulations.
Install, configure, and maintain observability and security components including Splunk ITSI, Enterprise Security, Edge Processor, and Ingest Processor.
Manage and optimize data pipelines and indexing strategies across Splunk and cloud-based storage layers (e.g., S3, Azure Data Lake, Security Lake).
Implement automation for lab provisioning, configuration, teardown, and orchestration using Terraform, Ansible, Python, and CI/CD pipelines.
Ensure system uptime, performance, RBAC compliance, and auditability across environments.
Leverage synthetic traffic tools and frameworks (e.g., Eventgen, Ostinato, Netropy, Attack Range) to simulate real-time telemetry and attack behaviors.
Develop and maintain documentation, environment diagrams, and reproducible deployment templates.
Act as a subject matter expert and mentor on Splunk platform architecture, performance tuning, storage, identity/RBAC, and data management.
Minimum Qualifications
Bachelor’s or Master’s degree in Computer Science, Information Technology, InfoSec, or a related technical field—or equivalent experience.
5+ years of experience in cloud infrastructure engineering, platform operations, or security/observability environments.
Deep experience with Splunk Cloud, Splunk Enterprise, and universal/heavy forwarders.
Hands-on knowledge of SPL (Search Processing Language) for dashboards, alerts, and reporting.
Strong understanding of observability and SIEM concepts, including metrics, logs, traces, correlation, and alerting.
Proficiency in cloud platforms (AWS, Azure, GCP), particularly with infrastructure-as-code, networking, and security configurations.
Proficiency in scripting and automation (Python, PowerShell, Bash) and configuration management tools (Terraform, Ansible, Puppet).
Experience managing data retention, indexing, storage optimization, and data model acceleration in Splunk.
Familiarity with identity management systems (e.g., Okta, Azure AD) for role-based access control and SSO integration.
Strong documentation, troubleshooting, and incident response skills.
Experience with cloud-native logging and data lake solutions like Amazon Security Lake, Azure Data Lake, and GCP’s Chronicle.
Familiarity with synthetic data generation tools and frameworks for attack simulation and telemetry emulation.
Experience contributing to field enablement initiatives, customer-facing lab scenarios, or partner solution architectures.
Relevant certifications (e.g., Splunk Core Certified Consultant, AWS Solutions Architect, Certified Kubernetes Administrator).
Splunk, a Cisco company, is an Equal Opportunity Employer and all qualified applicants will receive consideration for employment without regard to race, color, religion, gender, sexual orientation, national origin, genetic information, age, disability, veteran status, or any other legally protected basis.