Join us as we pursue our disruptive new vision to make machine data accessible, usable and valuable to everyone. We are a company filled with people who are passionate about our product and seek to deliver the best experience for our customers. At Splunk, we’re committed to our work, customers, having fun and most importantly to each other’s success. Learn more about Splunk careers and how you can become a part of our journey!

Role:

As a Principal DevOps Engineer, you will be responsible for supporting internal Splunk solution delivery related to IT operations and security use cases. The innovations built by our team will be widely shared and you should be eager to drive these efforts and share your knowledge with the Splunk community.

The Splunk at Splunk team is renowned for excellence in engineering, both inside and outside of Splunk. Last year, we presented five highly-rated sessions at .conf20, published multiple blog articles, and open-sourced several of our innovations via Splunk’s GitHub project, including but not limited to: ansible-role-for-splunk, docker-swarm-splunk-hf, and TA-VaultSync.

Splunk at Splunk is also a strong partner with the Products and Engineering teams, where we’ve contributed to everything from Splunk-built addons (e.g. Zoom, Gmail, Falcon Data Replicator), to the health checks and alerts that ship with Splunk’s Distributed Management Console (DMC), to the feature roadmap for Data Stream Processor (DSP). If you’re looking for a team where you can make a big impact, this is it!

Responsibilities:

  • Collaborate with the team to develop cutting-edge solutions for corporate data collection to Splunk Cloud. You will have the opportunity to work on projects that involve the latest technologies, such as serverless data collection using AWS Lambda and the development of microservices running on Kubernetes.
  • Seize opportunities to automate Splunk administration and deployment workflows using your knowledge of DevOps tools (Ansible, Terraform, GitLab CI/CD pipelines), Python, REST APIs, and Splunk's product portfolio.
  • Develop, test, and deploy a variety of Splunk configurations, Splunk technology add-ons, modular inputs, external lookups, and custom search commands using Python and Splunk SDKs.
  • Implement automated testing, continuous integration, and continuous deployment to streamline how we operate Splunk and data collection services internally.
  • Instrument infrastructure monitoring, application performance monitoring (APM), and tracing using Splunk Infrastructure Monitoring (SignalFx) combined with Ansible roles and Terraform providers for Splunk’s corporate IT services.
  • Implement security best practices related to our team’s Linux-based infrastructure, Python code, and Docker services.

Requirements: 

  • 12+ years working with distributed systems at scale in IT
  • 8+ years of software development experience in Python (expert level)
  • 5+ years DevOps experience in a technical lead role
  • 5+ years experience with configuration management (Ansible, Puppet) and infrastructure provisioning tools (Terraform, CloudFormation)
  • 5+ years experience with AWS, Docker containers, and container orchestration (Kubernetes, EKS, etc.)
  • 5+ years experience in cloud platform administration (preferably AWS)
  • Demonstrated ability to develop advanced CI/CD pipelines in GitLab (e.g. using dynamic child pipelines, automated unit testing, performance testing, etc.)
  • Strong knowledge of IT Security (encryption, firewalls, secure systems design, vulnerability management, secure software design, credential management, proxies, WAFs, etc.)
  • Deep understanding of IT fundamentals, including Linux and Microsoft operating systems, networking, security, cloud (AWS), and database technologies (MySQL, Postgres)
  • Excellent communication skills, both verbal and written; able to explain complex technical topics to both internal and external customers with ease
  • Fully proficient in git and version control systems (GitLab, BitBucket, GitHub)
  • Passionate about IT, technology, and following the latest industry trends

Preferred knowledge and experience: These are a huge plus.

  • Experience developing in Golang or writing custom Terraform providers
  • Experience developing custom Ansible modules
  • Certifications in AWS, Docker, Ansible, and/or Security (e.g. CISSP, CEH, Security+, etc.)
  • Published open-source projects on GitHub
  • Splunk experience a plus, but not required

Education: 

  • Bachelor’s degree or higher in Computer Science, Information Technology, Computer Information Systems, Information Security, or a closely related field
  • Master’s degree or higher in Computer Science, Information Technology, Computer Information Systems, Information Security, or a closely related field preferred but not required

What We Offer You: 

  • An opportunity to innovate cutting-edge solutions using Splunk software for Splunk itself. Plus, the ability to share those innovations with our customers! How cool is that?
  • A rare opportunity to join one of the best Splunk engineering teams in the world with a stable, collaborative, and supportive work environment that encourages work-life balance.
  • A constant stream of new things for you to learn. We're always expanding into new areas, bringing in open source projects and contributing back, and exploring new technologies.
  • A set of extraordinarily hardworking, innovative, open, fun and dedicated peers, all the way from engineering and QA to product management and customer support.
  • Growth and mentorship. We believe in growing engineers through ownership and leadership opportunities. We also believe mentors help both sides of the equation.

We don't expect people to work 12-hour days. We want you to have a successful time outside of work too. Want to work from home sometimes? No problem. We trust our colleagues to be responsible with their time and dedication and believe that balance helps cultivate an extraordinary environment

This isn’t a job – it’s a life changer – are you ready?

Splunk has been named one of San Francisco Bay Area’s “Best Places to Work” by the San Francisco Business Times, ten years in a row. We offer a highly competitive compensation package and a plethora of benefits.

We value diversity at our company. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, or any other applicable legally protected characteristics in the location in which the candidate is applying.

For job positions in San Francisco, CA, and other locations where required, we will consider for employment qualified applicants with arrest and conviction records.

 
Splunk's Hiring Practices
Splunk turns machine data into answers. Organizations use market-leading Splunk solutions with machine learning to solve their toughest IT, Internet of Things and security challenges.

Individuals seeking employment at Splunk are considered without regards to race, religion, color, national origin, ancestry, sex, gender, gender identity, gender expression, sexual orientation, marital status, age, physical or mental disability or medical condition (except where physical fitness is a valid occupational qualification), genetic information, veteran status, or any other consideration made unlawful by federal, state or local laws. Click here to review the US Department of Labor’s EEO is The Law notice. Please click here to review Splunk’s Affirmative Action Policy Statement.

Splunk also has policies in place to protect the personal information candidates disclose to us as part of the application process. Please click here to review Splunk’s Career Site Privacy Policy.

Splunk does not discriminate against employees or applicants because they have inquired about, discussed, or disclosed their own pay or the pay of another employee or applicant. Please click here to review Splunk’s Pay Transparency Nondiscrimination Provision. 
 
For job positions in San Francisco, CA, and other locations where required, we will consider for employment qualified applicants with arrest and conviction records.

Splunk is also committed to providing access to all individuals who are seeking information from our website. Any individual using assistive technology (such as a screen reader, Braille reader, etc.) who experiences difficulty accessing information on any part of Splunk’s website should send comments to accessiblecareers@splunk.com. Please include the nature of the accessibility problem and your e-mail or contact address. If the accessibility problem involves a particular page, the message should include the URL of that page.

Splunk doesn't accept unsolicited agency resumes and won't pay fees to any third-party agency or firm that doesn't have a signed agreement with Splunk.

To check on your application click here.
 
DIVE DEEPER
Find out what makes Splunk such a great place to work
Our Values

Splunkers are encouraged and empowered to be Innovative, Passionate, Disruptive, Open and Fun.
Learn More

Our Locations

From San Francisco to Shanghai, Splunkers work in 25+ offices across the globe.
Learn More

University Recruiting Program

Intern with people you want to hang out with, even outside the office.
Learn More

Our Blog

Hear from Splunkers on the latest.
Learn More

Diversity & Inclusion

Culture of Inclusion: Splunkers Share Their Stories
Learn More

LinkedIn

Follow Splunk on LinkedIn for job announcements, company news, and more.
Learn More