Splunk Base : SplunkAdministration : #3427

Sun T5120/T5220

araitz — Fri, 16 Oct 2009 12:23:19 PDT

To make sure this is clear:

Splunk is a multi-process, multi-threaded application. Indexing, as with most aspects of splunkd, is multi-process multi-threaded. However, there are some aspects of splunkd (namely input and compression/decompression) that are not multithreaded for obvious reasons. On T2000, these single-threaded components cause bottlenecking.

Sun T5120/T5220

erscott00 — Fri, 16 Oct 2009 11:10:04 PDT

Thanks Araitz & Rotten! I appreciate your feedback! I am going to play it safe and go with an x86 architecture since Splunk is a single threaded application.

Sun T5120/T5220

rotten — Fri, 16 Oct 2009 09:16:43 PDT

So I guess we have two problems with different metrics and capabilities:

1) How fast can you index the data?

2) How fast can you search that data?

Assuming I/O rates are comparable...

If I understand you, indexing is better done with a high clock speed, single threaded number crunching processor than with a 'wide' set of parallel processors. (Even when you have 2,000 sources feeding in from 100+ lightweight forwarders.) So if you want to look at Sun hardware to run your indexer, you might be better off with a IV+ CPU or maybe one of the new M-class servers than with a T-series.

Searching is better done with the 'wide' set of parallel optimized processors.

Since they run on the same server, and servers don't typically support both classes of processors at the same time, you should optimize for indexing because without the data there in the first place, searching won't do you much good.

Interesting. The user experience is based almost entirely on the search performance, and is usually the source of most performance related complaints that I've fielded at my current employer. Most of the users seem to be willing to live with a 5 minute indexing latency, but get grumpy when it takes forever for their search to complete. Lately I skewed the tunables on our pretty decently sized Intel/Linux server slightly in favor of searching, even still, some of our searches can take 5 or more minutes to complete. (Some of our reports can take 30+ minutes to process.). [ We can't use summary indexing for the reports because we are already at capacity for our license limit.]

I had always imagined that there was some cool parallel indexing going on. If you have multiple indexes would it speed up the indexing by making the indexer more parallel? I'd be willing to configure a separate index for every forwarder, and then set up the default searching to be across all of them if it would help both index and search performance by allowing greater parallization.

(I can't switch hardware types because of the aforementioned CIO's prejudice, so I'd still be on the same beefy linux box.)

Sun T5120/T5220

gkanapathy — Fri, 16 Oct 2009 08:39:35 PDT

Funny thing is, it sounds like this type of server would probably make a very good search head, since those (i.e., reads) are extremely parallelizable.

We haven't really benchmarked it for that though. I haven't heard of that side being the performance bottleneck. Because of the nature of Splunk workloads, it's the indexing (i.e., writes) that are the usually the main concern.

Sun T5120/T5220

araitz — Fri, 16 Oct 2009 07:09:47 PDT

Rotten,

Appreciate your perspective, but you should believe me on this one, I have plenty of first-hand experience with this architecture and Splunk. My educated opinion is that T2000/UltraSparc are very poor candidates for Splunk.

There are many problems with T2000 for what Splunk is trying to do:

- heavy abstraction - i.e. there are not 64 "cores" there are 8 cores that are abstracted 8 ways
- slow clock speed per physical core
- no math co-processor
- I could go on....

The point is that the things Splunk does that are single threaded, i.e. input, compression, etc can cause the T2000 to enter a state wherein because one splunkd thread is using 100% of its allocated CPU, the T2000 blocks other splunkd threads trying to get concurrent resources.

As I mentioned above, the T2000 is designed to run 16 JVM or 16 Apache servers concurrently. You can run multiple Splunk instances on the T2000 architecture and squeeze more out of them, but at the expense of manageability. Even then, performance still doesn't compare favorably with x86_64.

Check out this Sun blog post for more information:

http://blogs.sun.com/timc/entry/the_seduction_of_single_threaded

Sun T5120/T5220

rotten — Fri, 16 Oct 2009 06:48:59 PDT

I find this hard to believe (sorry!). Sun has several different classes of processors that perform differently under different conditions.

The T5120/5220 servers use the T2 processors which are exceptionally good at massive parallel transactional processing. I would think they would perform really well with Splunk.

The Intel chips are historically really good at floating point computations. My gut feeling is they would not scale as well as a T2 or T2+ chipset with Splunk.

The traditional line of "ultrasparcs" (III, III+, IV, IV+, etc...) are great for big databases, "deeper level" integer based processing, and single threaded apps. I can see where something like Splunk might suffer on one of the older processors in comparison to a modern Intel chip or a T2. On the other hand, these RISC chips really can outscale and outperform some of the other chips when you start running big relational database loads, data warehouses, or doing any sort of long running data processing.

I don't work for Sun, and the CIO at my current employer has some sort of deep seated personal grudge against them, so I haven't kept up with the very latest hardware developments. However, I would be really surprised if the 5120 performed only as good as any comparably priced Intel solution out there. It really should smoke those other systems -- especially in high throughput multi-threaded applications.

Sun has a performance lab, and they are usually pretty willing to work with vendors to benchmark products across their hardware suite. My understanding is they don't charge vendors for this service. It helps them sell hardware, after all, if your product will run faster on their systems than any of their competitors.

Here is more info on the Sun CPU types:
http://www.sun.com/products/microelectronics/products.jsp

Referring to them as all members of one class of CPU -- "ultrasparc" -- does them a great injustice.

Sun T5120/T5220

araitz — Thu, 15 Oct 2009 17:54:02 PDT

Yes, performance with Splunk on UltraSparc is very poor. UltraSparc are not designed for the heavy lifting that Splunk does in regard to processing. They are better candidates for Java, Apache, and other more lightweight applications wherein many instances are running concurrently.

Sun T5120/T5220

erscott00 — Thu, 15 Oct 2009 15:29:49 PDT

Hi All,

Has any one benchmark/implemented a Sun Microsystems T5120/T5220 using Splunk?

Any information is much appreciated...

Thank You,
RS