CLOUD

Dynamic Data: Data Retention Options in Splunk Cloud

Choice is always good! With that thought, we're happy to announce* a new subscription option for our Splunk Cloud customers called Dynamic Data: Active Archive, or DDAA for short.

If the name didn't give it away, DDAA builds on the Dynamic Data: Self-Storage (DDSS in short) feature we announced earlier in the year.

So why this new data management option? It's all about giving our customers flexibility and choice. In talking with customers, it became very clear that they wanted choices in how their data is managed in Splunk Cloud. DDSS is designed to provide a path for customers to self-manage data as it ages out. Customers are then responsible for restoring this data should the need arise to search against it. With DDAA on the other hand, Splunk manages the archival and restoration of data.

There are two key differences between the two capabilities:

  1. Data Management: With DDAA, Splunk provides complete lifecycle management of the archive on your behalf and remains the custodian of your data. Just like your Active Searchable data, Splunk manages all aspects of archive availability, durability, security and privacy requirements on your behalf. With DDSS, you the customer are responsible for data once it ages out. You define an Amazon S3 self-storage location and decide data from which indexes lands there. Once the data lands in self-storage, you're in complete control.
  2. Data Restore: DDAA enables you to request a slice of your data to be restored back into your Splunk Cloud instance. The entire workflow is fully integrated into Splunk Web so your archived data is available at your fingertips with predictable time between retrieval to search. With DDSS, if you want to search against the data stored in a self-storage location, you will need to restore it to a separate Splunk instance.

Dynamic Data: Active Archive is built on the same design principles like Dynamic Data: Self-Storage. Specifically:

  • Honor the data lifecycle: There is one copy of data in Splunk Cloud. When it reaches end of its useful life in Splunk Cloud based on retention settings in your control, you can choose to move the data to a storage location in your control or move it to Splunk managed archive. When data is moved out successfully, only then is it deleted from Splunk Cloud.
  • Secure and performant: Both capabilities are designed to move data with negligible impact to your routine search activities. We’ve incorporated best-security practices using AWS IAM roles

Let us look under the hood and learn more about how DDAA works:

Dynamic Data Active Archive is an optional service. Once you subscribe to the service, you will notice a few changes to the index listing page. 

 

 

Storage Type now has a new value, Splunk Archive. For indexes that roll over into the archive, a new Restore option is available; more on that later. 

For an index, you can now choose Splunk Archive, in addition moving the data to Self-Storage. Note that these options are mutually exclusive, i.e. for an index you can either chose Archive or Self Storage NOT both. For an index, it is possible however to switch between the options.

If you select the Archive option you can specify the Retention for that archive. The retention is based on the entitlement you signed up for when you subscribed to the service.

Once the options are set, as the Size or Searchable time criteria is met, the data is rolled into Spunk Archive. As mentioned earlier only when the data is successfully moved to the archive, is it then deleted from Splunk Cloud. That's it! 

The day though will arrive, when you will be asked to restore data from the archive for an incident investigation or to meet a compliance request. With 4 simple clicks you can easily restore the data from the archive into your Splunk Cloud instance.

 

Simply specify the time slice, give a description, check the size and you're all set! If you want to notify people once the data restore is complete, you can specify their email id’s. You can also look at the history of restore requests for that index and see details like status, data volume restored, etc.

A couple of key points about data restore:

  • We’ve designed restoring up to 10% of your archival data entitlement at any given time into the subscription price. For example, let us assume that a customer with 1000 GB daily ingestion, signs up for the service on January 1st, 2019. Annually, they will archive 365 TB. They are entitled to restore up to 3.5TB of data. 
  • Once the data is restored, it is available in your Splunk Cloud instance for 30 days, after which it is deleted automatically. If you do not need to hold the data for 30 days, you can delete it manually by clicking Clear under the Actions column in the Restore Archive window.

Once the data is restored in your Splunk cloud instance, you can search it like any other event data! The team ran successful beta programs for both the capabilities and we were fortunate to have many Splunk customers participate, giving us invaluable and overwhelmingly positive feedback.

The team also presented a session at .conf18 and got great positive feedback from the attendees and throughout the conference. If you missed it, catch the replay of "Your Data Your Way: Data Retention Choices in Splunk Cloud."

At Splunk we value customer feedback and continually look to deliver innovations that meet and exceed our customers’ expectations. Dynamic Data Self Storage and Dynamic Data Active Archive are examples of successful collaboration with our customers! 

*The availability of Dynamic Data Active Archive capability is planned for Q1 2019.

Manish Jiandani is Director of Solutions Marketing for Business Analytics and IoT at Splunk. Prior to Splunk he was a was co-founder at Falkonry, an innovative provider of condition monitoring solutions for industrial systems based on advanced machine learning techniques. As a product management professional, Manish has spent 15+ years successfully building and launching market leading products in BI, Analytics and Hadoop.

Join the Discussion