Splunk-o-ween. A Machine Data Horror Story


It was a dark and stormy night at ACME Corporation. An IT guy, let’s call him Bill and his Product Manager, Ted (most excellent names) are sat in the office on the evening of Thursday 30st October. In a strange twist of fate the ACME UK office is an old mansion set on top of a big hill surrounded by nothing but ancient woods being battered by gale force winds and a most heinous storm. The time is 11:59 PM and Bill and Ted are trying to get a release live in time for the weekend. With power cuts, no mobile phone signal and a problem somewhere in either the code, the production environment or the database, things aren’t going well. They’d looked at database logs, restarted servers, rolled back to previous versions – nothing.

With a huge clap of thunder and a flash of lightning and the lights going out again, the clock turned midnight. Bill slammed his coffee cup down on the table (he was six espressos in to the release). “Ted – this is bogus” he growled. “I don’t know where this thing is going wrong”. No reply. Bill stood up and knocked a whole load of empty pizza boxes over on to his keyboard. He stood up and went looking for Ted by the light of his mobile phone. The office was pitch black with an eerie glow from the odd monitor that was on standby. As Bill cursed his way through the support team’s office and the fact they had left their window open, there was another flash of lightning. In front of him was a six foot armoured figure looming over him. Using all his training from a weekend’s paintballing, Bill grabbed one of the support team’s NERF guns and fired at the figure. A hollow, empty sound came from the metal plated spectre as it loomed closer…

It was when the lightning flashed again that Bill realised that the figure was the life sized cardboard mockup of Boba Fett that the support guys kept next to the door to the machine room that was blowing in the wind due to the window being open.

Ted was nowhere to be found in the support room. The wind and rain provided a noisy background which hid the sounds of the machine room as Bill walked past. With another flash of lightning, Bill saw a figure cowering in the corner of the machine room, facing away from the door, shaking.

Blair Witch

The figure was bathed in a green light from the racks and blades. As Bill ducked down behind the machine room window, all he could hear over the storm was a mixture of insane cackling mixed with whimpering and strange chanting (best described as the sound the offspring Gollum from the Lord of the Rings and the Wicked Witch from the Wizard of Oz would make if they’d inhaled one too many helium balloons) such as:

“ooooh yes, you want to know that the errors”

“must be ready by tomorrow – hmmm”

“I’ll tell you what the customers have done if you can write a Perl script”

Bill considered his options. Ted was his friend but this strange goblin creature was in the machine room, the release still had to be ready and he was kind of armed with a NERF gun. Bill stood up, took a deep breath and slammed his shoulder against the door. It was then he realised that the door was locked and he hadn’t entered the code on the keypad. Fumbling, he tried to enter the door code of 666 (it is a Halloween story!) The ghoulish figure bathed in the green light ran to the door and pressed his contorted face to the glass as Bill somehow continued to fumble with three simple numbers whilst holding a NERF gun and stopping Boba Fett from falling on him. There was a flash of lightning and Bill was face to face with the horror on the other side of the glass…

The twisted, blood smeared face of Ted was staring back at him. Bill screamed. Everything went silent. Ted pressed his face against the glass. It was then Bill realised it wasn’t blood, it was tomato sauce from the pizzas they ordered. Ted’s garlic and oregano breath steamed the glass as he whispered “I’ve done it”. Bill whispered back “what have you done? And why are you whispering?”

Ted backed away from the glass, twisting his body as if he’d had an electric shock (it turns out that this was his celebration dance). He picked something up from the floor and ran back to Bill “look – the answer” he screeched. (Clearly Ted had been on the coffee as well as the pizza.) Bill warily looked at what Ted had picked up – his laptop. On it were a series of hieroglyphics and runes (an analytics dashboard) – “I’ve found it” said Ted. Bill looked confused. “THERE” Ted shrieked, pointing to what turned out to be a line on a graph. The lights flickered again as the storm worsened. There’s the problem. Someone changed the database connection pool size from 40 to 4”.

Bill continued to look confused through the glass and was increasingly concerned about the unnatural acts, sacrifices or deals with the devil that Ted must have done to get the answer “What black magic is this? What have you done?”

Ted wiped the pizza “blood” from his face “it isn’t black magic you fool – it is something called Splunk. I took all the information, put it into Splunk and traced down the error across everything we’ve got for the release”.

That’s when the power came on and the phone rang. Bill answered the phone on a nearby desk and Ted answered the phone inside the machine room. A voice on the other end said “Hi, this is Rufus from the security team, it looks like we may have a security attack and you’re the only ones left in the office”.

Bill and Ted’s screams were drowned out by another clap of thunder. That’s when the real horror began…

Splunk_Pumpkin 2

(Thanks to Glenn Block for the Splunk-o-ween name)


Posted by