Principal Splunk Engineer, Splunk@Splunk - US Remote

Join us as we pursue our disruptive new vision to make machine data accessible, usable and valuable to everyone. We are a company filled with people who are passionate about our product and seek to deliver the best experience for our customers. At Splunk, we’re committed to our work, customers, having fun and most importantly to each other’s success. Learn more about Splunk careers and how you can become a part of our journey!

Role:

The Splunk at Splunk team leverages all elements of Splunk’s portfolio to create solutions for Splunk’s internal IT and Security organizations. We use a combination of Splunk Cloud, Enterprise Security, ITSI, VictorOps, SignalFX, and DSP as key ingredients in our solutions.

This team is renowned for excellence in engineering, both inside and outside of Splunk. Last year, we presented five highly-rated sessions at .conf20, published multiple blog articles, and open-sourced several of our innovations via Splunk’s GitHub project, including but not limited to: ansible-role-for-splunk, docker-swarm-splunk-hf, and TA-VaultSync.

Splunk at Splunk is also a strong partner with the Products and Engineering teams, where we’ve contributed to everything from Splunk-built addons (e.g. Zoom, Gmail, Falcon Data Replicator), to the health checks and alerts that ship with Splunk’s Distributed Management Console (DMC), to the feature roadmap for Data Stream Processor (DSP). If you’re looking for a team where you can make a big impact, this is it!

As a Principal Splunk Engineer, you will be responsible for all aspects of Splunk solutions that related to IT operations and security use cases. The innovations built by our team will be widely shared and you should be eager to drive these efforts and share your knowledge with the Splunk community.

Responsibilities:

• Collaborate with the team to develop cutting-edge solutions for corporate data collection to Splunk Cloud. You will have the opportunity to work on projects that involve the latest technologies, such as serverless data collection using AWS Lambda and the development of microservices running on Kubernetes.
• Seize opportunities to automate Splunk administration and deployment workflows using your knowledge of DevOps tools (Ansible, Terraform, GitLab CI/CD pipelines), Python, REST APIs, and Splunk's product portfolio.
• Develop, test, and deploy a variety of Splunk configurations, Splunk technology add-ons, modular inputs, external lookups, and custom search commands using Python and Splunk SDKs.
• Implement automated testing, continuous integration, and continuous deployment to streamline how we operate Splunk and data collection services internally.
• Instrument infrastructure monitoring, application performance monitoring (APM), and tracing using Splunk Infrastructure Monitoring (SignalFx) combined with Ansible roles and Terraform providers for Splunk’s corporate IT services.
• Implement security best practices related to our team’s Linux-based infrastructure, Python code, and Docker services.

Requirements: 

• 12+ years working with distributed systems at scale in IT (with some experience in service operations preferred)
• 5+ years of software development experience in Python (expert level)
• 5+ years of DevOps experience (Ansible, GitLab CI/CD, Terraform, Docker, Kubernetes)
• 5+ years performing Splunk administration, architecture, and development
• Strong knowledge of IT Security (encryption, firewalls, secure systems design, vulnerability management, secure software design, credential management, proxies, WAFs, etc.)
• Deep understanding of IT fundamentals, including Linux and Microsoft operating systems, networking, security, cloud (AWS), and database technologies (MySQL, Postgres)
• Excellent communication skills, both verbal and written; able to explain complex technical topics to both internal and external customers with ease
• A solid technical understanding of Splunk products
• Fully proficient in git and version control systems (GitLab, BitBucket, GitHub)
• Passionate about IT, technology, and following the latest industry trends

Preferred knowledge and experience: These are a huge plus.

• Experience with Docker, Kubernetes, and AWS is highly preferred
• Experience building CI/CD pipelines
• Experience administering Linux systems
• Experience integrating systems with Hashicorp Vault
• Experience with load balancing technologies
• Experience developing content in Splunk ITSI a plus but not required
• Certifications in Splunk, AWS, and/or Docker

Education: 

• Bachelor’s degree or higher in Computer Science, Information Technology, Computer Information Systems, Information Security, or a closely related field

What We Offer You: 

• An opportunity to innovate cutting-edge solutions using Splunk software for Splunk itself. Plus, the ability to share those innovations with our customers! How cool is that?
• A rare opportunity to join one of the best Splunk engineering teams in the world with a stable, collaborative, and supportive work environment that encourages work-life balance.
• A constant stream of new things for you to learn. We're always expanding into new areas, bringing in open source projects and contributing back, and exploring new technologies.
• A set of extraordinarily hardworking, innovative, open, fun and dedicated peers, all the way from engineering and QA to product management and customer support.
• Growth and mentorship. We believe in growing engineers through ownership and leadership opportunities. We also believe mentors help both sides of the equation.

We don't expect people to work 12-hour days. We want you to have a successful time outside of work too. Want to work from home sometimes? No problem. We trust our colleagues to be responsible with their time and dedication and believe that balance helps cultivate an extraordinary environment

This isn’t a job – it’s a life changer – are you ready?

Splunk has been named one of San Francisco Bay Area’s “Best Places to Work” by the San Francisco Business Times, ten years in a row. We offer a highly competitive compensation package and a plethora of benefits.

We value diversity at our company. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, or any other applicable legally protected characteristics in the location in which the candidate is applying.

For job positions in San Francisco, CA, and other locations where required, we will consider for employment qualified applicants with arrest and conviction records.