Letters from a Splunk Admin

No one writes letters anymore.  It’s been such a long time since I’ve written a letter, it got me thinking what I would even write about… which then got me thinking what would a Splunk Admin write a letter about?  No, not really, but I needed an intro for this blog.

So if your awesome Splunk Admin were to write a letter, it might go something like this…

Hello Beautiful Splunker,

I am contacting you in regards to your scheduled search usage in Splunk.
Currently, when you log into Splunk, you may be greeted by this fabulous warning bar:

Would you kindly review the scheduled searches you have created?  Go to Manager > Searches and reports.  Under the Owner drop down menu, please select your name.  For each saved search which has been scheduled, please examine the following:

   - Start and finish times
   - Scheduled run times
   - Expiration times <--- please pay special attention to this one

We have noticed there a lot of reports generated every 5 minutes (*/5 * * * *) and kept for 7 days.

  1440 minutes in a day / 5 minutes = 288 reports per day
  288 reports * 7 days = 2016 reports per week

--> Yes, that is 2016 reports retained for just one search running every 5 minutes with a 7 day retention. <--

Splunk will try its best to keep a maximum of 2000 searches by default and will delete the older ones, but it cannot delete search artifacts which have not reached their expiration age.

If you need help in determining the appropriate schedule and expiration period, we can help.  Please contact us.  To get you started, below are best practices when scheduling searches/reports.

   1. Run the search manually and see how long it takes (let's say it
      takes 8 minutes to run).
   2. Never schedule the search on a faster interval than it takes for
      the search to run (if it takes 8 minutes, don't run it every 5
      minutes, set the manual cron to be */8 * * * *).
   3. Do not schedule the search to run across overlapping time spans
      (usually, running a search every 8 minutes looking across data
      over the last 24 hours is not necessary).
   4. If you email yourself the results and never plan to visit the
      Splunk URL to view them, please delete the results before the
      next run (set the expiration to a custom time, 8 minutes).
   5. If you are using cron syntax, make sure it is what you intend.
        0 0 */3 * *  means run every 3 days
        * * */3 * *  means run every minute of every hour every 3rd day
      And yes, this has been known to happen.

Here is an example of a well-mannered scheduled search:

   1. Set the search to run for the previous day.  More on modifers here.

   2. Run the search everyday at 1 am (during off peak hours).

   3. Email myself the results and delete them after 12 hours.

Thank You,
Your Distinguished Splunk Admin

I’m guessing your Splunk Admin might be too busy to write letters, but now you don’t have to wait.  Get inspired.  Don’t be a milquetoast.  Wrangle those runaway scheduled searches!

Thank you to one of my best customers and Splunk Admins, Sean W., who inspired this post and actually wrote the first draft of this letter to his Splunk friends.  People do still write letters!

Vi Ly

Posted by