TIPS & TRICKS

Packaging Splunk

Splunk runs on a lot of platforms for a relatively young product and that number is always increasing. The day I started, there were packages for Intel and PowerPC Macintoshes, i686 Linux, Solaris 8 on Sparc, and FreeBSD on x86, all created with BitRock InstallBuilder, run from a simple shell script, usually by Erik. There really wasn’t much control over what went into the installer — if a file was in the installer prep directory and the shell script didn’t know to delete it, out it went.

By the time 2.1 was on its way, we’d decided to switch to native packages, and our list of platforms had expanded to include Solaris on Intel, with several more on the horizon. We also wanted to provide the “rail tarball” distribution we continue to support, in part so that QA could get started before the packaging automation was complete.

What is that packaging automation, you might ask? Obviously writing custom code to package each platform (not to mention spec or pkgmap files in each platform’s native format) was not a very maintainable solution. Instead we use a locally modified version of Easy Software’s EPM package manager. After a little work, EPM lets us use a common set of list files to create relocatable packages using common pre- and post-install scripts across all of the 9 platforms we now build on. We’re able to control every file and permission that goes into the packages, and in most cases we can add packaging for a new OS platform with a minimum of work (for something very different we haven’t previously had in house, like AIX, more time might need to be spent cleaning up EPM’s support for the platform). We’ve piggy-backed creation of the “rail tarball” distributions on the EPM list file structure, so those packages too are completely defined. EPM itself is built during the Splunk build process like any other 3rd party dependency, so any new patch to the tool are available to the build systems almost as soon as it’s checked in.

The downside to this is a little loss of flexibility for some platforms; trivial changes to the RPM spec file or FreeBSD ports originpath have usually required modifying EPM’s source. It’s not a bad trade-off, though.

EPM has recently been made open source; check it out at http://www.epm-home.org . If you’re interested in my patches, feel free to drop me a line, but in some cases superior patches have been contributed at epm-home.org.

Splunk
Posted by

Splunk

Join the Discussion