By Splunk April 29, 2014
Guess what? Your veteran Splunk administrator took a new job and you were designated as his replacement! Since you are brand new to Splunk and there is no budget for training, you decide to download Splunk to your laptop and explore this new search solution. You load some logs in the blink of an eye and are searching and building dashboards in minutes. You download apps and think to yourself, “No problem… I got this! I can read the documentation and take those Splunk Administration classes later.” Beaming with confidence you have your IT department install a brand new Splunk production server with one CPU and four gigabytes of RAM in VMware and skip right past the Splunk Distributed Deployment Overview documentation. After all, one server is more than enough right? You start adding more and more data, touting the virtues of Splunk to anyone that will listen. Soon afterwards people from all over your organization start pinging you, asking if you can put their application data and logs into Splunk. You teach people the basics and send them on their merry way, happy that they too can now evangelize the virtues of searching and reporting with Splunk. People are having fun generating time saving reports and scheduling keyword searches for the word “error” that runs every five minutes over all time. And one very smart guy decided to save his same scheduled search five different times, in each of the apps, just so he could get to them easier. Life is good?
A few weeks later you start getting trouble ticket calls about your Splunk server. People are asking why their searches are sooooo slow, why their data is missing, why events are corrupted, and why their logs are time-stamped with a different time than what shows in the actual logs. You log back into Splunk and run a quick search, but an alert message pops up saying, “Error in ‘UnifiedSearch’: Your Splunk license expired or you have exceeded your license limit too many times. Renew your Splunk license by visiting http://www.splunk.com/store or calling 866.GET.SPLUNK.” You remember that your company bought a license last year and they were only indexing operating system logs at the time. In your short tenure as a Splunk administrator you can proudly say that you installed tons of the free apps from apps.splunk.com including the Splunk App for Windows Infrastructure, Exchange, Unix, and the Splunk Cisco Security Suite. But even with all of your best efforts, people are complaining because they just can’t search any more. You are desperate for answers and everyone is depending on you to make everything work again. Maybe setting up your production Splunk server environment with no training wasn’t such a good idea after all. Life is not so good.
Desperate for help, you turn to Splunk and the numerous support options available to you. With a huge community of Splunk users on the answers.splunk.com portal, a real-time Internet Relay Chat (IRC) room called #splunk, and the Splunk Technical Support organization you are confident that you can resolve all of these search issues very quickly. You open a technical support ticket only to find that you are not an authorized support portal user. You quickly reach out to your Splunk sales contact, are made a Splunk support portal admin, receive a license reset key, and get set up for a meeting to talk about getting an additional license for the increased log volume.
Now that a support case is officially open you receive an email asking you to run a, “splunk diag” on the server and upload the diagnostics information to your support case. You have no idea what a “splunk diag” is and it doesn’t matter because you don’t have direct access to the Splunk server to run that diagnostic command anyway. Finally, after another day of waiting you finally have the right permissions to access to the Splunk server. If only you knew how to navigate through a Linux server. What does $SPLUNK_HOME mean again? I guess you better go find the Unix admin to assist you. Soon thereafter your system administrator helps you run the proper Splunk commands and you upload the diagnostic files to your support case. You quickly receive a detailed response from your friendly neighborhood Splunk technical support team and they help you resolve all of your configuration issues. All will be okay! Hooray!
Looking back you realize that if you put some discipline and process around your Splunk implementation from the very beginning you would have much better capabilities and operational intelligence in place today. Invest some time up front referencing the arsenal of tools, documentation, educational material, and support infrastructure available to you before you dive head first into one of the most powerful search tools in the industry. All of our available resources are here: http://www.splunk.com/resources. Happy Splunking!
P.S. If you feel like you are still overwhelmed and need some additional help, feel free to purchase a Splunk Advisory Services Package. Splunk Advisory Services Packages deliver Splunk expertise designed to ensure successful deployments. These services are backed by Splunk experts, ensuring consistent and quality delivery, architecture, training, and ongoing sustainment for Splunk in your enterprise.
----------------------------------------------------
Thanks!
David Maislin