INTERNET OF THINGS INTERNET OF THINGS

How to Stream Internet of Things Data into Splunk in Ten Easy Steps!

Inspired by Discovered Intelligence’s blog post “How to Stream Twitter into Splunk in 10 Simple Steps” last week, I began thinking about a simple Internet of Things example where we could demonstrate an easy integration of IoT platforms and data into Splunk that everyone could access. There are plenty of great examples of Internet of Things services we could poll with the REST modular input, but for this example we will be building a quick integration with LogMeIn’s Xively service using their “Test Drive” tutorial.

This walkthrough assumes you have a working install of Splunk 6 and have installed the REST API Modular Input App. If not, please follow the links and you should be up and running in no time.

Xively’s Test Drive tutorial is advertised as “a 5 minute tutorial to get familiar with all the basics from connecting one device to building interconnected systems and apps”. I can’t really argue with that based on my experience. So here it is, connecting Splunk to Xively’s Internet of Things Platform in 10 Simple Steps!

  1. Go to https://xively.com/testdrive/ and sign up or log in (see security note below).
  2. Click “Take Test Drive” According to Xively, this tutorial is designed to simulate a package tracker using your mobile phone for the tracker.
  3. Read the “lets make a package tracker” section and click “Next”
  4. Create a Xively feed to store your device’s data by going to “develop” (under “Web Tools” in the upper right). Call it “My Phone” and set its Privacy Settings to “Private”.
  5. Copy down the Feed ID, API Key, and API Endpoint ID as you will need them later. Click “Next”
  6. Connect your phone to Xively by entering the Feed ID and the API Key into the form. Click “Send the Link”.
  7. In Splunk, add a new REST Input by going to Settings->Data inputs. Click REST to add a new input.
  8. Click “New” and name your new input whatever you would like (I chose the ultra-creative “Xively”).
  9. Configure the input to use the following settings and save your input:
    • Endpoint url: the endpoint id you copied from your device page on Xively
    • HTTP Method: GET
    • Authentication Type: basic
    • Authentication User: your Xively username
    • Authentication Password: your Xively password
    • Response Type: JSON
    • Sourcetype: xively feed
  10. Open the “Here’s your tutorial” link from Xively on your phone. Start shaking your phone frantically. (Ok, I had one extra step because I assumed you already had the REST app installed. Kudos to Discovered Intelligence for getting the Twitter Setup done in 10 steps including REST App installation!)

splunkxively2

Data should begin appearing in Splunk within 60 seconds. If you want faster updates you can set the poll interval property in the inputs settings to something smaller, like 15 seconds. To begin searching your data, move your phone around (with the tutorial link open) and run the following search from the Splunk search app with a timerange of real-time (5 minute window):

 sourcetype="xively feed"

Every minute or so, a new JSON event should arrive that starts like this (datastream array is expanded):

{[-]
   created:2014-01-21T15:59:51.255145Z
   creator:https://xively.com/users/yourusername
   datastreams:[[-]
{[-]
      at:2014-01-21T21:47:59.631768Z
      current_value:0
      id:shake
      max_value:3.0
      min_value:0.0
}
{[-]…

Shake your phone a bit, and monitor the changes for the array with “id:shake”, click the “Arrived” button on the webpage and see the new array with “id:arrived” populate.

If you really want to go crazy, run the following search and view it in a line chart:

sourcetype="xively feed"|rename datastreams{}.current_value as current_value|rename datastreams{}.id as id|eval x=mvzip(id,current_value)|mvexpand x|eval x=split(x,",")|eval id = mvindex(x,0)|eval current_value = mvindex(x,1)|timechart span=1m latest(current_value) by id

splunkxively

And thanks to Splunk Answers for that search: http://answers.splunk.com/answers/70843/stats-function-on-json-data, a quick google search for “splunk stats JSON” had me up and running!

I would also suggest creating a new Xively log in for this exercise, as using basic authentication is probably not the most secure method. I’ll be taking a look at trying to improve this walkthrough by creating an oAuth app as well as trying out the streaming endpoints but if you beat me to it please let me know! I’ll also eventually take a look at building a Splunk Data Model around the JSON structure and will update when that is complete. Maybe I’ll even create a Splunk for Xively app.

Potential uses for this type of integration are limited only to your imagination. Moving beyond this phone based demo to real world IoT devices, you could use this same method to Splunk data from your Xively connected IoT devices to troubleshoot performance or better understand device data like battery use. You could also use Splunk to develop algorithms for your IoT devices to allow them to interact more efficiently with their environment and improve end user experience by monitoring devices for uptime and user input errors. And you could even use Splunk’s new integrated Maps and geolocation features to analyze your data based on geography. Finally, these concepts are not limited to Xively alone, there are likely many IoT PaaS out there with REST API endpoints you could access and understand with Splunk and the REST API Modular Input App.

Take advantage of Splunk’s powerful analytics and search language to perform predictive analytics of sensor data and device performance from large numbers of connected devices, do cross device or system data correlation, and use Splunk’s time based search commands to better understand cause and effect with your deployed devices and sensors!

I’d love to hear what other cool things you might be doing with Splunk and IoT PaaS services like Xively, please drop me a note at bgilmore@splunk.com, find me on Twitter at @BrianMGilmore, or on LinkedIn at www.linkedin.com/in/industrialdata. Hope to hear from you soon!

Brian Gilmore
Posted by Brian Gilmore

Join the Discussion