More fishbucket fun

For debugging files getting re-indexed, sometimes what I want to see can only be found in the fishbucket index of the affected instance. I can pick up and move an entire index (3.x+) and drop it into another instance, but when working with the fishbucket there are a couple other things to watch out for. I don’t want anything to change it once I put it in the new instance. So I set up a throwaway instance to easily make changes I wouldn’t want to do to a real one.

REALLY BIG WARNING

Don’t do this to any Splunk instance you like. You will be unhappy later. Throw away your dummy instance when you are done so you don’t confuse anybody.

Set up a new instance of an appropriate version, the same or more recent as the original and appropriate architecture (ppc/sparc or intel.) Get it all working with the correct ports so you don’t conflict with anything else that may be running on the machine. Since it won’t be indexing, the license doesn’t matter. Start and then stop so the first run stuff is done.

Change some things so it won’t touch the index:
./splunk clean all -f
rm /opt/splunk/bin/splunk_optimize
rm /opt/splunk/etc/system/default/inputs.conf (or wherever it is in your version)
edit /opt/splunk/etc/system/default/indexes.conf to comment out the line frozenTimePeriodInSecs = 2419200 in [_thefishbucket] stanza
If it’s large, you’ll want to also comment out maxDataSize = 10

rm -rf /opt/splunk/var/lib/splunk/fishbucket/*
copy the contents of the fishbucket index you have into the now empty directory (don’t accidentally create an extra fishbucket/fishbucket directory!)
remove any archives or other temporary files you left lying around in the index directories

Start this instance and now you can search for index=_thefishbucket. It helps to exclude the Splunk internal files with something like this:

index=_thefishbucket NOT filename::/opt/splunk/var/log/splunk/license_audit.log NOT filename::/opt/splunk/var/log/splunk/metrics.log NOT filename::/opt/splunk/var/log/splunk/searchhistory.log NOT filename::/opt/splunk/var/log/splunk/splunkd.log NOT filename::/opt/splunk/var/log/splunk/splunklogger.log NOT filename::/opt/splunk/var/log/splunk/web_access.log NOT filename::/opt/splunk/var/log/splunk/web_service.log

Your full path may vary. What is left is all the files being monitored by the instance.