In part 1 of this blog series we spoke about how to turn sensor data into logistics insights. In this part we outline one approach for anomaly detection and enrich our sensor data with location information to discover geospatial patterns.
Anomalies? Find them with a few lines of SPL.
Anomaly detection can be tricky and implementations vary from simple thresholding and baselining to highly sophisticated approaches based on machine learning. In this example we leveraged the Splunk Machine Learning Toolkit to detect numeric outliers using a sliding window approach to check against multiples of the standard deviation in this time series to spot anomalies.
And that’s how the SPL looks like:
| timechart span=1s avg(ax) as avx avg(ay) as avy avg(az) as avz
| eval sum=abs(avz-60.0)+abs(avy-6.0)+abs(avx-282.0)
| fields - avx avy avz
| streamstats window=100 current=true avg(sum) as avg stdev(sum) as stdev
| eval lowerBound=(avg-stdev*2), upperBound=(avg+stdev*2)
| eval anomaly=if(sum < lowerBound OR sum > upperBound, 128, 0)
| fields _time sum anomaly
The results here reflect pretty well what we wanted to achieve. Big impacts in relative circumstances are flagged as an anomaly and would indicate severe effects on our deliverable. We could further fine tune this model, cross validate, check with reality again and operationalize, but for a first quick analysis the results look quite meaningful.
More meaning? Add context information – like geolocation data.
What if we can enrich our raw sensor data with other data to make it even more valuable? In our example we merged in geolocation data to add geospatial insights. This allows us to know where anomalies happened and we can drill down into a heat map to further explore spatial patterns of our anomalies.
This visualization provides us with immediate insights about possible geospatial patterns of our anomalies. Did the shocks occur frequently in the same locations, say a crossroads or areas of delivery? These kind of questions can now easily be explored, analyzed and actually answered with real data. Different business decisions can be based on this data, e.g. change routes to prevent damage from reoccurring due to certain local conditions.
Wrap up: Make sensor data actionable
Ok, let’s sum up what we have reached so far. We enriched sensor data with geolocation information to gain insights about our delivery process. This allows us to draw actionable, data-driven business decisions to optimize our processes.
Let’s say we further enrich our data with information from our delivery network of our service providers to give us performance monitoring of the quality of services we engage. This provides us with detailed insights about the quality of goods delivery and helps us to define KPIs and metrics that may be useful for contract designs or claim for damages. As all data is gathered and analyzed in real time we can leverage sensor data for immediate business decisions.
Thank you for reading.