Happy New Year! I hope it was relaxing, restful and slightly over-indulgent. If you’re anything like me, you’re probably wondering if you can stick to any New Year’s resolutions unlike last year (less chocolate, more exercise, less time glued to my mobile phone, better jokes, improved punctuation etc.) So far this year (six days in) I’ve managed to succeed in one resolution – catching up on reading – and it got me thinking about some “Big Data New Year’s Resolutions”.
1) Use Big Data Applications to give you something new. We’ve got all this big data, what are we doing with it? Data scientists can use it to find answers, customer service/experience teams can get a better view and understanding of what we’re buying and IT can use the machine data to ensure operational effectiveness. What about developers? My colleague, Brett Sheppard was interviewed on the subject recently (you can read it here) My first big data new year’s resolution is to look at the applications we’re going to be able to build in 2014 now we have better access to all the data.
2) Consider how analytics for contemporary generation data solutions offer something different from data warehouses. My second New Year’s resolution is to look at what is possible now with analytics and data visualization in the newer generation of data platforms. The compelling things to consider are the price/performance, the time to delivery and crucially the unstructured nature of the data that we need to work with. This could be analytics on top of Hadoop but also technologies such as NoSQL, In-Memory etc. I finally got to read articles from the summer that I’d added to my reading list (6 months old I know – clearly I failed in 2013’s New Year’s resolution to read more!). The one article that summarised it best for me was this one.
Obviously there are horses for courses and data warehouses still have their part to play. However, the business analytics capabilities of platforms such as Hunk, Datameer and others means that data warehouses aren’t the only race in town any more…
3) Mine the insight your machine data will give you into how your business processes really work. In a previous life I spent many years working in middleware and BPM. One of the challenges was always how to get started in identifying how things “really worked”. This was sometimes described as process discovery. A lot of this information is already captured. Your log files probably contain a very high percentage of a customer’s behavior across multiple channels, both business and technical. This information remains untapped in a lot of cases but probably describes a number of your business processes. I spent some time with a large UK retailer before Christmas. They thought they had modeled their customer processes. When they compared this with the data they were already capturing from their core application log files – they saw the real process was very different, more complex and less streamlined. They appear to have had a very successful Christmas season for sales and positive customer experience having understood their customer “journey” better.
4) Treat App Management and IT Operations as a big data problem. The estate of technology most organizations have to manage isn’t going to get considerably smaller in 2014. It might become simpler with an increase in cloud-based infrastructure, platforms and software as a service. We’re still going to be under pressure from the V’s of big data even if we see more demand for velocity and slightly less growth in volume as we become more real-time in the actions we have to take. The role of managing applications and ensuring effective IT operations has already got a lot of benefit by adopting some of big data approaches and tools and 2014 feels like a year where this will take another jump. I enjoyed reading Tom Fisher’s blog post here. With BugSense and Cloudmeter becoming part of Splunk – I’m excited to see what benefits customers are going to be able to get by having the holistic view of machine data now including network and mobile device data available to IT, DevOps and support.
5) Look for the sweet spots where analytics, IoT and big data converge. My colleague, Brian Gilmore, spent some time with the guys at ABI research before Christmas and they are doing some really interesting research into analytics and IoT. If the Internet of Things delivers 10% of what it promises then we’re going to be generating a lot of data that could give us some hugely valuable analytics into our behavior, the environment, health, security, safety, lifestyle etc. There are going to be some fascinating use cases we’ve only just scratched the surface of and some predictions suggest that it could spark a new industrial revolution.
6) Security looks like being a place that’s going to drive innovation, proactivity and disruption. I wouldn’t claim to be a security expert but 2013 seems like a year where everything changed and could be a time that we look back on for many years to come. We’ve seen a lot of interesting use cases for Splunk and security. Most obvious is our role in the SIEM space and how security required big data thinking and the ability to correlate events from huge amounts of fast moving, variable data. You may have seen the announcement that Splunk partner, FireEye acquired Mandiant. What struck me was the disruptive “bricks and mortar” approach of actually sending in people to fix the security breaches that the technology identifies.
7) A bit of imagination and the spirit of the “garage inventor” will show us something new. Excluding how Santa Claus is using Big Data – 2014 will be a year where we’ll see people start to combine the big data that’s available with their imagination and the spirit of invention. Consistently we see the first person who “gets” big data and what’s possible with it become a very popular person inside an organization. Being able to answer questions you haven’t been able to before gets people’s imagination going. Questions such as “can you tell me what this customer has done” or “when does this happen most” or “do we have information on xxx?” often trigger a company to think in different ways. The phrase “the art of the possible” is probably overused now but when you can see the kinds of value and solutions that can be created in a much shorted timescale than ever before – it does push back the boundaries of what a company considers creating or inventing themselves. You can see this spirit of “garage inventor” and using imagination in a couple of examples – the NHS and New York Air Brake. To quote the New York Air Brake example:
“… Splunk customer New York Air Brake’s (NYAB’s) Train Dynamic Systems Division has harnessed operational intelligence by collecting large volumes of data from freight train locomotives and braking systems. The system calculates and analyzes forces occurring on each of the couplings between freight cars in real time to present an optimized driving strategy to the engineer. Using Splunk for real-time data analysis allows them to notify key personnel about system and driver anomalies, present customized analytics dashboards that give operational insight, and continuously improve optimization algorithms. NYAB is providing insight that eliminates inefficient physics in the operation of those trains and has a potential to save customers a billion dollars or more in fuel costs a year.”
Good luck sticking to your own resolutions and hopefully the above gives you food for thought (just not too much food if you’re trying to keep to a losing weight New Year’s resolution!)