By Splunk December 04, 2011
The funny thing about Splunk is how it just doesn’t stop surprising you. Even after years of using it, you still get surprises. Okay, I must confess I haven’t used Splunk for years, but you get the idea.
Last week, I was in the land of Kimchis in a -5 degrees Celsius (23 degrees Fahrenheit) room, wearing no gloves and a thin veil as my underwear. It was brutal as usual, especially when you are not in your own country. I was thumbing thru the Splunk User’s manual trying to look for some answers. Out of the corner of my eye, I saw a few paragraphs that were new to me. In fact, there were so new that I believed they were only added into the manual from 4.2.3 onwards. Heck, nobody actually mentioned anything about them, but hang on, it’s probably because I am in Kimchi land.
In case you are wondering, real-time backfill has nothing to do with “Summary Indexing” backfill, the latter some of you may already be familiar with. According to the documentation, this is what real-time backfill does:
For real-time windowed searches, you can specify that Splunk backfill the initial window with historical data. This is run as a single search, just in two phases: first, a search on historical data to backfill events; then, a normal real-time search. Real-time backfill ensures that real-time dashboards seeded with data on actual visualizations and statistical metrics over time periods are accurate from the start.
You can enable real-time backfill in limits.conf in the [realtime] stanza:
[realtime]
default_backfill =
* Specifies if windowed real-time searches should backfill events
* Defaults to true
This feature literally freed my partner and allow them to achieve a very important requirement in their app – to provide historial backfills in their real-time dashboards. Before that, they were using all sorts of scheduled savedsearches to achieve similar results, and it was porky at best.
If you remember my earlier blog on the Splunk at F5 ASEAN User Conference, I was working on the integration of a live Unstructured Supplementary Service Data (USSD) feed into Splunk, and was demoing this to a live audience. Unfortunately this feature was not available back then, and it made testing of the App somewhat a bit more difficult as we need to wait for the live data to stream in before you can be sure that the dashboards you were building are going accordingly to what you wanted.
This is a fantastic feature to have.
----------------------------------------------------
Thanks!
Tat-Wee