By Splunk August 14, 2012
If you follow the Tech Blogs, you’ve likely heard of Dalton Caldwell‘s App.net, which is attempting to become the user and developer funded social network glue for the next generation of social services. Because I’m an insufferable early adopter, I ponied up the $100 for a dev account and went to town this weekend trying to figure out how best to get social data out of App.net into Splunk. We have pretty extensive experience here working with Twitter data, and since App.net data is also social data and looks similar, I was pretty confident we could provide value.
First, we needed the data in Splunk. App.net has been in development for about a month, so there’s no firehose implemented yet. However, they do have a REST endpoint for a global feed. I manually walked through the OAuth 2.0 flow to get an access code for App.net. Once I had that, it was relatively trivial to write a quick python script to query the Global timeline every X seconds (currently set to 10), log the raw JSON to a file, and then have Splunk pickup the file. You can see the project on GitHub with the code for the logger.
Of course, there are only about 60K posts on the network in total right now, so the absolute numbers are hardly impressive, but the amount of progress on the network in just the last 4 days is hard to believe. You can see a list of the applications being developed for the network, and there are already mobile apps, new web front ends, statistics applications, etc for the network.
With the data in Splunk, I began crafting some dashboards to visualize the traffic on App.net. You can see some of the early cuts included in-line here. However, I wanted to share the data I was getting back besides simply taking screenshots and posting them to imgur. So, I decided to follow the example we had put on dev.splunk.com to send the data to Leftronic. I’ve seen Leftronic before, and it’s a really cool dashboarding technology, and it’s already public and scalable which served my needs.
I put together the code on this GitHub project, based on the aforementioned example. It runs a number of real-time searches against Splunk and then makes REST API calls to Leftronic to update the dashboard. I then shared out the dashboard via a simple iframe’d web page at appnetstats.com. The result is above. It’s proven very popular on the network, and hopefully will serve as an inspiration for other weekend project ideas.
----------------------------------------------------
Thanks!
Clint Sharp