TIPS & TRICKS

SplunkIt v2.0.2 Results & EC2 Storage Comparisons

With a new version of SplunkIt out the door, it was time to get new benchmark numbers, both for EC2 and the commodity hardware we tested in previous posts.  Now that SplunkIt is compatible with Windows and Splunk 5, we can get a good picture of how Splunk will perform across different operating systems and major Splunk versions.  Grab the new kit here.

All tests used a search user (splunkit-user) on a VMware ESX 4.1 instance with 1 Xeon 2.67GHz core, 2GB RAM, Linux 64bit, and Firefox 9.  All posted bonnie++ IOPS numbers were gathered by invoking bonnie++ in the following fashion, for those who are interested:

bonnie++ -d <PATH> -s <RAM*10> -qf -u root -n 50

In these first examples, we’ll see how your choice of EC2 storage may affect Splunk search & index performance.  We evaluated 3 different storage options: ephemeral instance storage, elastic block storage (EBS), and EBS with 800 provisioned IOPS.

As the results show, EBS provides a noticeable gain in search & index performance over ephemeral instance storage, on top of the redundancy that already comes with EBS.  Provisioned IOPS increases indexing performance even more, and should improve search performance as well if running many concurrent searches.  Standard EBS also appears to provide inconsistent I/O performance compared to provisioned IOPS, as the range of bonnie numbers & Amazon’s EBS documentation suggests.

Linux on EC2: c1.xlarge

Splunk Version Storage Bonnie IOPS Avg KBPS Avg EPS Avg First Event (sec) Avg Search (sec)
4.3.6 Instance storage 139 7857.41 27672.71 2.14 34.31
4.3.6 EBS 90-220 8113.15 28572.76 2.47 34.71
4.3.6 EBS, 800 prov. IOPS 833 9225.93 32562.43 2.23 34.73
5.0.2 Instance storage 139 9674.22 34110.03 2.17 26.57
5.0.2 EBS 90-220 10649.2 37574.35 3.02 26.5
5.0.2 EBS, 800 prov. IOPS 833 12410.5 43639.83 2.12 27.37

System: EC2, c1.xlarge; CPU: 8 vCPU, 20 ECU; Memory: 7GB; OS: Linux 64-bit, Ubuntu 12.04LTS

Windows on EC2: c1.xlarge

Splunk Version Storage Bonnie IOPS Avg KBPS Avg EPS Avg First Event (sec) Avg Search (sec)
4.3.6 Instance storage 139 3835.05 13485.25 2.7 56.51
4.3.6 EBS 90-220 5355.24 18877.92 2.81 56.36
5.0.2 Instance storage 139 3209.51 11297.69 2.86 41.95
5.0.2 EBS 90-220 5324.7 18744.26 3.04 42.8

System: EC2, c1.xlarge; CPU: 8 vCPU, 20 ECU; Memory: 7GB; OS: Windows Server 2008 R2 64-bit

Linux on EC2: other instance types

System CPU Memory Splunk Version Avg KBPS Avg EPS Avg First Event (sec) Avg Search (sec)
EC2, m1.xlarge 4 vCPU, 8 ECU 15GB 4.3.6 8348.67 29466.64 2.85 39.78
EC2, m1.xlarge 4 vCPU, 8 ECU 15GB 5.0.4 10043.79 35252.17 3.08 32.47
EC2, m2.4xlarge 8 vCPU, 26 ECU 68GB 4.3.6 10972.87 38722.4 2.75 26.89
EC2, m2.4xlarge 8 vCPU, 26 ECU 68GB 5.0.4 12642.68 44515.19 2.65 20.4

Storage: EBS; Bonnie IOPS: 90-220; OS: Linux 64-bit, Ubuntu 12.04LTS

These results highlight the substantial performance improvements introduced in Splunk 5 across linux and windows, especially using Elastic Block Storage (EBS) in EC2.  Down the road we may experiment with the hi1.4xlarge SSD instances & other EC2 optimizations to study their impact on searching & indexing.

Bare Metal: RAID 0 and RAID 1+0

We revisited the lab system we tested in our first results blog post, this time with SplunkIt v2.0.2.  We tested the kit with Splunk installed on RAID-0 and RAID-10 (1+0) partitions, with results posted below.

Indexing & search performance was comparable between the two, with RAID-10 providing a slight improvement in search times.  The mismatch in the number of disks – 6 in RAID-10 case, 12 in the RAID-0 case – may explain why RAID-10 didn’t outperform RAID-0 in all indexing tests.  We would expect a larger improvement in search performance from RAID-0 to RAID-10 if our data was spread across many more buckets.  Like the EC2 tests above, this test highlights the benefits of upgrading to Splunk 5.

Splunk Version Storage Bonnie IOPS Avg KBPS Avg EPS Avg First Event (sec) Avg Search (sec)
4.3.6 RAID 0, 12 15K RPM Disks 738 21557.97 76015.55 2.51 27.36
4.3.6 RAID 1+0, 6 15K RPM Disks 733 21812.66 77975.81 2.29 27.21
5.0.3 RAID 0, 12 15K RPM Disks 738 22709.03 80141.51 2.6 20.48
5.0.3 RAID 1+0, 6 15K RPM Disks 733 22400.7 79057.94 2.48 20.18

System: HP DL380G7; CPU: 2×6 Xeon 2.67GHz; Memory: 12GB; OS: Linux 64-bit, Fedora 14

We’ll periodically post SplunkIt results for new configurations and hardware.  Feel free to send us questions, feedback, and your own test results at splunkit@splunk.com.

Jason Beyers
Posted by

Jason Beyers

Join the Discussion