Steps for manual migration to Splunk 4.x

Steps for manual migration to Splunk 4.x

This topic describes the procedure for migrating a 3.x Splunk installation to version 4.x manually. Before you proceed, be sure to review "What to expect when migrating to 4.0" in this manual.

If you have not customized your 3.x deployment extensively, you may be able to use the automatic "in-place" migration process described in the Migrate on Windows or Migrate on UNIX topics. In particular, if you have not customized deployment.conf, you can likely use the automatic migration process.

Some custom configurations cannot be migrated at all, and must be rebuilt within the new Splunk framework. Refer to the section titled "What cannot be migrated" in this topic and "What to expect when migrating to 4.0" in this manual for more details.

Share your migration experiences

Did you migrate an existing Splunk installation to 4.x and run into some issues? What led to your success? Share your tips with other users on the Splunk Community Wiki "Migration experiences" page.

Overview of the manual migration process

During the manual procedure, you will make a backup copy of your existing deployment, install Splunk 4.x in a separate path from 3.x, and then migrate your data and configuration files across from your 3.x deployment. This procedure is designed to allow for a minimum of downtime.

It's important to follow this procedure carefully--Splunk 4.x differs significantly from version 3.x, and many configuration files cannot simply be copied over. Splunk provides scripts for converting and migrating the content of these files so that they will function correctly in 4.x.

Read through this entire topic before beginning the procedure. This way, you will know what to expect, and can plan accordingly.

What cannot be migrated

As mentioned earlier, Splunk 4.x is significantly different from earlier versions. Some aspects of your existing deployment cannot be migrated, and must be rebuilt.

Splunk Web display customizations

All things to do with Splunk Web have been completely re-architected and rebuilt from the ground up. As a result, any customizations you've made that affect the display of Splunk Web cannot be migrated; you must rebuild them using the 4.x architecture and tools. These include:

• Form searches
• Field actions (field_actions.conf)
• Dashboards - visit here for instructions on how to rebuild them
• UI preferences (prefs.conf)
• Report chart and table preferences on saved searches
• Changes to UI strings (literals.conf)
• 3.x Apps - read this Developer Manual topic for guidance on reworking them.

Considerations for deployment server and forwarders

• If you use the Splunk deployment server, you must rebuild your deployment configuration using the new deployment server architecture in 4.x; it cannot be migrated. If you've made changes to deployment.conf in version 3.x, you should not copy it over into your new Splunk 4.x installation. Set it aside to use as a basis for building your new deployment server configuration.
• If you have deployed forwarders, you can delay migrating them to 4.x; 3.x forwarders will work with a 4.x version of Splunk. This is especially useful for large forwarder deployments. You can wait until you are comfortable with your new deployment server configuration before migrating forwarders to 4.x.
• Version 4.x forwarders do not work with version 3.x deployment server.

Before beginning the migration

Before you begin migrating your Splunk data and configuration files, make a backup copy of your 3.x installation. The easiest way to do this is to run the Splunk CLI diag utility, which will create a compressed (.tar) file containing your entire installation. To do this, from $SPLUNK_HOME/bin type:

On UNIX:

./splunk diag

On Windows:

splunk diag

If you have difficultly running diag in your environment, you can also run the python script directly using the cmd command:

./splunk cmd python /opt/splunk/lib/python2.5/site-packages/splunk/clilib/info_gather.py

This produces splunk-diag.tar.gz|zip that you can preserve as a backup of your 3.x deployment.

Once you have done this, use the instructions in this manual to install version 4.0 for your platform in a different location from your existing 3.x install but do not start it yet.

Do not start your 4.x Splunk instance yet.

Import configuration files that do not require manual migration

Some of the configuration files from your 3.x deployment can be copied over to your new 4.x install without any migration required.

Remember to copy these files from/to $SPLUNK_HOME/etc/system/local or from your custom config directory--do not copy over the files in /default!

• Copy over all configuration files except deployment.conf.
• Edit the copy of inputs.conf you put on your new 4.x Splunk instance to set all the inputs to disabled=true.

Import user and password information

You can bring your user data, password, and other authentication information across at this point:

• authentication.conf
• authorize.conf
• splunk.secret
• passwd file

Do not start your 4.x Splunk install yet.

Migrate your Splunk index data

This section explains how to configure your new 4.x install to read your existing Splunk index data. Data indexed by version 3.x is completely compatible with 4.x and does not need to be re-indexed. However, note that it may have been indexed in a less efficient form, so searches performed over data that was originally indexed and migrated from 3.x may not be as fast as searches over newly-indexed 4.x data.

Again, be sure to follow these instructions carefully to minimize downtime and data loss. Order is important! Additionally, be prepared to perform these steps in rapid succession. Once you roll the 'hot' index data directory to 'warm' on your existing Splunk deployment, you'll want to immediately move on to the next steps as quickly as possible to ensure there is a minimum of downtime in the transition to indexing with your new 4.x Splunk install.

To understand the implications of "hot" and "warm" index directories, review the "Best practices for backing up" topic in the Splunk Community Wiki.

1. In the root directory of your 4.x install, edit splunk-launch.conf to point the value of SPLUNK_DB to point to the index data directories of your 3.x deployment. By default, this value is /opt/splunk/var/lib/splunk, but you can check the splunk-launch.conf in your 3.x deployment to be sure you've got the right path.

2. Then, in your version 3.x deployment, use the CLI to force a roll of any data in your hot index data directory to the warm' directory. To do this, type:

• ./splunk search '| oldsearch !++cmd++::roll' -auth splunk
  • You'll see an error about "Search Execute failed because Hot db rolled out to warm" right afterward; you can safely ignore it. You'll also need to provide the admin password to execute this CLI command.
  • To roll any other indexes you are currently writing to, use:
  • ./splunk search ' | oldsearch index=<INDEX_NAME> !++cmd++::roll' -auth admin:<ADMIN_PASSWORD>

3. As soon as you've rolled all your indexes from hot to warm, immediately shut down your 3.x deployment and start your 4.x install up. When you start your 4.x, some of the configuration files you copied over may undergo an auto-migration process. Splunk will display information about this process when you start 4.x for the first time.

4. As quickly as you can, log into your new 4.x install and search for data you know is in your 3.x index, to validate all the migration steps. Ensure that your new deployment is able to search and find data in your 3.x indexes. If the validation fails, immediately restart your 3.x deployment before proceeding with any troubleshooting.

5. Once you have successfully validated that your 4.x deployment can search your 3.x index data, immediately edit inputs.conf and set your inputs to disabled=false and restart your 4.x Splunk deployment to enable them. Verify that data is coming into your new deployment.

Now you're ready to begin using your new 4.x Splunk deployment!

If you use distributed search

If you have configured distributed search, you must distribute new keys to all search peers to use distributed search in 4.x. These keys are generated at startup if distributed search is enabled.

The keys will be generated in $SPLUNK_HOME/etc/auth/distServerKeys/. Distribute the files $SPLUNK_HOME/etc/auth/distServerKeys/trusted.pem and private.pem from one host to any others distributed search peers.