That happened: episode 3

This week in “That happened: notes from #splunk”, a blog about the goings-on in the Splunk IRC channel:

Trying to grep your desk?

Some problems even Splunk can’t solve:

<mchesmo3> splunk please organize my life…..
<DaGryph> There’s an app for that.
<jspears> mchesmo3: splunk doesn’t enforce a schema, it just makes the piles easier to find stuff in
<jspears> “search, don’t sort”

It’s gonna be the future soon*

Data from the future is a pretty common symptom of timestamp recognition problems, but maybe:

<cerby> something is horribly horribly wrong with data I’m getting in splunk.
<cerby> The index_status_health showing all time for buckets tells me I have data up to year 2038 in several dozen buckets.
<^Brian^> wrong with the data or wrong with how splunk is processing the data?
<^Brian^> cerby: check the timestamps..it may be processed wrong :)
<cerby> ^B – unknown.
<pie|home> ooh future data
* ^Brian^ has had that happen alot
<pie|home> yeah, i bet it’s timestamp recognition
<Drainy> why does everyone assume its a timestamping issue, perhaps the Splunk instance has gone a step futher and has started Splunking events *before* they happen
<Drainy> Splunk> Ahead of its time, and yours
<^Brian^> heh
<^Brian^> Splunk> Seeing sh*t before you do :)

* http://www.youtube.com/watch?v=LDiDK_yBCw0

Cuckoo for cut

Bonus points if you can figure out what Nerf was trying to do and post it in the comments (#splunk denizens who were present in the channel at the time of this exchange are disqualified :)):

<@hexx0> Nerf:  try this
<@hexx0>for i in `$SPLUNK_HOME/bin/splunk cmd btool indexes list | grep 'homePath = volume:splunk_db' | cut -f2-3 -d "/"`; do SZ=`du -ms $SPLUNK_HOME/var/lib/splunk/$i | cut -f1` ; TOTAL=`expr $TOTAL + $SZ` ; echo $TOTAL; done
<@hexx0> (I expect to be admonished by the awk Gestapo for my excessive use of “cut” any minute now)
<@amrit|wrk> wtf
<@amrit|wrk> hexx0 you crazy
<Nerf> You don’t need grep either 😉
<duckfez> however, a true awk studmuffin could have rolled the cut and greps into one command
<@hexx0> amrit|wrk: crazy for Ghetto Bash Scripting
<@hexx0> agreed, agreed

Rockstar dad

Helping you tag your authentication failures after the kids have gone to bed:

<Ayn> tinsley: tag::eventtype=authfail earliest=-30d@d | timechart span=1d count by host | search count>5
<Ayn> ?
<Ayn> tinsley: hmm, on a second though…that won’t work
<tinsley> Ayn: you’re right :)
<Ayn> tinsley: try this then:
<Ayn> tag::eventtype=authfail earliest=-30d@d | bucket _time span=1d | stats count by _time,host | search count>5
<tinsley> Ayn: you are unbelievable
<Ayn> i take it it worked! 😀
<tinsley> Ayn: yes, thanks so much! your splunk-fu is much better than mine
<duckfez> tinsley: yes, Ayn is pretty damn awesome
<Ayn> awesome. i just managed to put my daughter to sleep, my splunk-fu tends to increase after that
<Ayn> earlier on today it took me 2 hours to compose a 10-line mail…

rachel perkins

Posted by


Join the Discussion