Note: Do not attempt to migrate from 2.0.x to 3.0.x. You will first need to upgrade and migrate your data to 2.2.3 format. Read the 2.0.x to 2.2.x migration instructions first.*
Step 1: Administrative preparation Obtain your new license(s)Splunk 3.0 requires an entirely new form of license key. If you are a Splunk Professional customer, you must obtain a new Enterprise license (Professional is now known as Enterprise) even if your 2.x license has not expired. New licenses can be obtained through customer support or via your store account. If you have a current Enterprise support agreement, it is likely that customer support has already re-issued your license and attached it to your store account. Just log into www.splunk.com, and go to store -> my orders to view all of your licenses. Please contact support if this is not the case.
If you are using a Free license then there will be no need to install a 3.x license. The default 3.0 Free license will be installed automatically if no license is detected on startup.
Back up your old instance(s)Please backup your 2.x instances before attempting to update them to 3.0.1 or higher. At a minimum make sure the $SPLUNK_HOME/etc and $SPLUNK_HOME/var directories in all your instances are backed up to a separate location before proceeding. If you have Splunk instances in production we recommended piloting the update in a staging environment before attempting it in production. It is possible to recover a corrupt instance or one in an indeterminate state but you will require assistance from customer support to do so.
Plan your updatePlease read through both the upgrade overview and this entire page before attempting your first update. The update process involves overlaying Splunk 3.0.1 or higher (3.0.x) over your 2.x instance. If you're using packages native to your platform you'll use their update mechanisms. If you have a tar installation you'll simply extract 3.0.x over your 2.x instance after backing up your configuration. In either case you'll update your database to function with 3.0.x and move your configuration back in place after the 3.0.x package is in place. The time required to complete the update depends on the complexity of your configuration, not the size of your database(s).
You are not alonePlease do not hesitate to contact support if you have questions or experience problems updating to 3.0.x.
Step 2: Install the 3.0.x package and update your databaseNOTE: If you have a Splunk Enterprise license (formerly known as Splunk Professional) be sure you have obtained a new 3.0 license before proceeding.
This process requires the installed configuration to be moved out of the way and then be restored after installation and database migration. Until configuration is restored the default ports will be used. Those ports are 8000 and 8089. It would be best to ensure there are no conflicts on ports 8000 and 8089 before executing the data migration step. Note that Splunk 3.0.1 only listens on one HTTP port and one management port, unlike 2.x which listened on one HTTP port, one HTTPS port, and one management port. That is why only 8000 and 8089 are important for this step. The HTTP port can be configured to be HTTPS (see Step 3).
To install Splunk 3.0.x over 2.1.x or 2.2.x and fashion the 2.1x or 2.2.x database for use in 3.0.x:
Most configuration options previously controlled by XML files have now been moved to configuration files. This change makes it easier to
administer a Splunk server and easier to deploy configuration changes to other Splunk servers via bundles and the Splunk deployment server.
The purpose of this section is to provide a map between 2.x and 3.0.x configuration. It is not to provide exhaustive documentation of all
3.0.1 configuration. Please review the 3.0.1 administration guide for full details of how 3.0.1 works and 3.0.1 administration.
The process of updating and restoring your configuration involves moving some configuration information out of XML files and into bundle
files and updating your bundle files to work with 3.0.x. You'll need to migrate configuration parameters from $SPLUNK_HOME/etc.bak to $SPLUNK_HOME/etc.
At a high level the necessary configuration changes are:
By default Splunk 2.x opened three ports. An HTTP port, and HTTPs port, and a management port. Splunk 3.0.x only opens two ports. A web port and a management port. The web port can be configured to be either HTTP or HTTPS. The default 3.0.x configuration can be observed at $SPLUNK_HOME/etc/bundles/default/web.conf. Documentation and an example of the 3.0.x web.conf file can be found at $SPLUNK_HOME/etc/bundles/README/web.conf.[spec|example] or here.
To migrate your settings from 2.x to 3.0.1 it should just be a matter of configuring the same ports used in the 2.x search.user.xml file in an override 3.0.x stanza in $SPLUNK_HOME/etc/bundles/local/web.conf. If you're using SSL and are also using your own certificates then you'll want to place those certificates from your 2.x instance in the location where the default 3.0.x web.conf file expects them, or override those configuration parameters in your local web.conf file as well.
Multiple indexes in 3.0.xIn 2.x all indexes were specified in $SPLUNK_HOME/etc/myinstall/pluginConfs/multiIndexer.xml. This file has been converted into a bundle in 3.0.x. In 3.0.x the set of default indexes are configured in $SPLUNK_HOME/etc/bundles/default/indexes.conf. Additional custom indexes may be added in $SPLUNK_HOME/etc/bundles/local/indexes.conf. Be sure to place the 3.0.x configuration information after migrating your database. If you don't also configure your custom indexes in Splunk 3.0.x then you won't see them. It should be readily apparent how the parameters in the 2.x XML file maps to the 3.0.x bundle file. Be aware that the index dropdown in 2.x is not present in 3.0.x. Search in your custom indexes with the index::yourcustomindexname search operator.
Documentation and an example of the 3.0.x indexes.conf file can be found at $SPLUNK_HOME/etc/bundles/README/indexes.conf.[spec|example] or here.
Bundle Configuration Updates 2.x regexes.conf is now 3.0.x transforms.confThe file regexes.conf is ignored in 3.0.x. The file was renamed and extended to better serve its purpose - transforming data inputs and events based on requirements to modify and extend Splunk's automated processing. Like regexes.conf, transforms.conf is referenced by props.conf and may contain regular expressions to extract the target of the transformation. The format and actions of the individual attributes within the file has not changed. Rename your regexes.conf to transforms.conf. Then modify props.conf to refer to the transform.
Change regexes to transformsProps.conf used to refer to regexes in the regexes.conf file. Now it refers to transforms in the transforms.conf file. In addition, the attribute prefix in props.conf has changed from REGEXES- to TRANSFORMS-. See the 3.0.x admin manual reference pages on props.conf and transforms.conf for full details.
An example of Splunk 2.x to Splunk 3.0.x regex to transforms changes:
In props.conf change from the following 2.x style:
[cisco_syslog] MAX_TIMESTAMP_LOOKAHEAD = 32 SHOULD_LINEMERGE = False REGEXES = syslog-host ...
to the following 3.0.1 props.conf style:
[cisco_syslog] MAX_TIMESTAMP_LOOKAHEAD = 32 SHOULD_LINEMERGE = False TRANSFORMS = syslog-host ...
A stanza in a 3.0.1 transforms.conf looks like stanzas in 2.x regexes.conf:
[syslog-host] DEST_KEY = MetaData:Host REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?)\.\w+\s+)*\[?(\w[\w\.\-]+)\]?\s FORMAT = host::$1
Note that if you are using a regexes.conf stanza in 2.x in order to extract fields at search time for use with the report:: search modifier, you will want to read about how to define extracted fields in 3.0.1 as well as read about the new search language which has many powerful native statistical and structured search commands including select, {{where, {{fields, stats, top and rare which have replaced and improved upon the 2.x report:: search modifier.
2.x savedsplunks.conf and livesplunks.conf is now 3.0.x savedsearches.confThe 2.x savedsplunks.conf and livesplunks.conf files have been combined into one overall savedsearches.conf file. In 3.0.x you can add scheduling and alert information directly to the saved search. The old live splunk subsystem in 2.x has been completely replaced with the new scheduling and alerting subsystem in 3.0.x.
The name/value pairs in savedsplunks.conf should map directly to savedsearches.conf with one exception. Use just the raw search string in 3.0.1, not the entire XML value of the query parameter in 2.x. Be sure a user exists in the 3.0.x instance with the same userid you're bringing over from 2.x.
The name/value pairs from livesplunks.conf will not map cleanly into savedsearches.conf. You do not need to bring over the savedsplunkid parameter as alert information is now stored directly with the saved search. The next change is that the 3.0.x saved searches.conf file uses a cron-like scheduling parameter in replacement of several run and range parameters in livesplunks.conf. It should be readily apparent how to map the relation and action configuration if one compares your livesplunks.conf stanza to the 3.0.x spec and example files. ($SPLUNK_HOME/etc/bundles/README/savedsearches.conf.[spec|example])
Special note about saved report:: searchesThe only search syntax element that is not backwards compatible between 3.0.x and prior versions is report::. If you have saved searches that use report::, you should update them to take advantage of the new search language which has many powerful native statistical and structured search commands including select, {{where, {{fields, stats, top and rare which have replaced and improved upon the 2.x report:: search modifier. These new commands are both more flexible and faster than the old report:: modifier.
2.x auth.conf in LDAP mode to 3.0.x auth.conf in LDAP modeIf you're using LDAP authentication in 2.x then you can copy your auth.conf file into 3.0.x and use it if you make the following changes:
If you've modified your segmenters in your 2.x instance you should add them to your local segmenters.conf file. $SPLUNK_HOME/etc/bundles/local/segmenters.conf. See $SPLUNK_HOME/etc/bundles/README/segmenters.conf.[spec|example] for detailed information.
Data InputsIn general nearly all data input parameters should map cleanly from 2.x to 3.0.x with the exception of the regexes/transforms transition described above. If you have a concern about a particular parameter you should browse $SPLUNK_HOME/etc/bundles/README for the parameter in question and see if it's usage has changed from 2.x to 3.0.x. If something about the data input configuration gets lost in translation there should be clear error messages in splunkd.log. Be aware that 3.0.x is capable of eating archive files directly without needing them to be uncompressed first.
Other Configuration Updates Splunk UsersThe procedure for migrating non-LDAP users is covered in Step 2. The procedure for migrating LDAP configuration is covered in Step 3. Select the method that is appropriate for you.
Lightweight ForwardersSplunk 2.x required extensive configuration changes to run in a minimal mode for a forwarding-only instance. In Splunk 3.0.x this configuration is done for you automatically if you enable forwarding and disable local indexing in the GUI. Splunk should only consume about 100 MB RAM in the 3.0.x configuration, usually less. It is possible for 2.x forwarders to forward to a 3.0.x instance.
Custom C++ ProcessorsIf your 2.x instance contains a custom C++ module, that module should work with 3.0.x. Be aware, however, that Splunk 3.0 ships with fewer shared objects than Splunk 2.x. In particular, libstdc++ is no longer included with the distribution. If you use your platform's libstdc++ and other libraries, your module should work.
Distributed SearchIt is easiest to simply re-configure your 2.x distributed search hosts via Splunk Web in 3.0.x. Be aware that it is not possible to mix 2.x and 3.0.x servers in a distributed search cluster. Enhancements to the search language in the 3.0 product prevent this from working. Note that the "Splunk-2-Splunk" tab in the 2.x admin section has been renamed "Distributed" in the 3.0.x admin section.
Host tags and source type aliasesIf in Step 2 you exported your global data to an XML file you can convert and re-import the host tag and sourcetype aliases into your Splunk 3.0 instance. With $SPLUNK_HOME/bin/setSplunkEnv sourced from a 3.0 instance, execute
python $SPLUNK_HOME/bin/migrate_2x_exported_data_to_3x.py $YOUREXPORTEDFILENAME splunk import globaldata $YOUREXPORTEDFILENAME.readyfor30import -auth admin:$YOURPASSWORD
To confirm the procedure you should be able to see your host tags in type ahead and also see the correct data exported with a splunk export globaldata command.
Comments
Please confirm that you downloaded 3.0.1 or 3.0.2. There was no migrate_2x_data_to_3x.py in 3.0
Posted by m@ on Aug 11 2008, 10:16am
python $SPLUNK_HOME/bin/migrate_2x_data_to_3x.py. does not exist.
Posted by bc_unixadm on Aug 11 2008, 10:09am