The venerable old-skool Splunk forums are now closed. Feel free to search for old content here, but new posts are no longer supported.

Instead, please visit the thriving community at answers.splunk.com to ask and answer questions about your Splunk deployment and how to get the most out of it.

Forums: SplunkReporting: Monitoring custom performance monitors

Previous Topic: Server Downtime  |   Next Topic: BUG: Exporting "Show source" keeps going back to show 25 lines at most even if you selected another


Posts 1–4 of 4
1
posted by: tnine  |  posts
date: July 5, 2009
permalink

Hi all,
I'm evaluating Splunk and have a few questions.

1. Our main server is running on CentOs and most our our application clients will be on windows. Since all these nodes mirror one another, can I install the splunk forwarder on a single node then copy the configurations to each machine? I'm using splunk forwarders with no web interface.

2. Our applications have custom performance counters in them. I've found this post which outlines how to dump data from performance counters to file.

http://www.splunk.com/base/Documentation/3.4.10/Admin/WindowsProcessMonitoring

I've decided to use tab delimited in my output format. Is there any way to assign a human readable label to each column? For instance, mine will have 3 numeric values which represent failed tries, average delta between operations, and average operations per second.

Thanks,
Todd

[Revised on Sun, 05 Jul 2009 18:55:21 -0700]

Forgot to ask. Is it possible to define the file observations from the web interface?

2
posted by: araitz  |  posts
date: July 5, 2009
permalink

1) You will probably want to use Deployment Server:

http://www.splunk.com/base/Documentation/3.4.10/Admin/HowTheDeploymentServerWorks

2) You can use the DELIMS property of transforms.conf:

http://www.splunk.com/base/Documentation/3.4.10/Admin/Transformsconf

3) Yes, but you will have more options (including wildcards, whitelist/blacklist, etc) from conf files

3
posted by: gkanapathy  |  posts
date: July 5, 2009
permalink

Also, if your application is writing Windows Performance Counters, it's probably easier and cleaner to log using WMI, and you don't need to dump anything to a file.

http://www.splunk.com/base/Documentation/latest/Admin/WindowsManagementInterfaceWMIInput

And no, you can't delete indexed from the Splunk Web interface.

4
posted by: tnine  |  posts
date: July 11, 2009
permalink

Thanks for the feedback guys. I've unfortunately already given WMI a try, but it doesn't return the calculated value correctly. For instance, my operations/sec one is calculated by (currentValue-lastvalue)/(currenttime-lasttime). When I use WMI to query the value I'm only getting the "currentValue" from my query, not the calculated value I need. When I use perfmon to configure file output, the output in the file is the calculated value. Here's what I've set up so far.

My host has Windows Splunk Forwarder to our linux box. It doesn't index anything, only forwards data. The forwarder has the following settings.

Full Path:C:\perflogs
Source Type: Manual tsv (For tab separated values)

Then on my linux box I have the following.

in transforms.conf
<pre>
#Windows tab seperated values example value: "07/12/2009 17:43:33.485" "0" "2" "0.13333452729976311"

[windows_perf_tsv_fail]
REGEX = "(.*)"\t"(\d+)"\t"(\d+)"\t"(.*)"
FORMAT = Number of consecutive failed operations::$2

[windows_perf_tsv_delay]
REGEX = "(.*)"\t"(\d+)"\t"(\d+)"\t"(.*)"
FORMAT = Number of consecutive failed operations::$3

[windows_perf_tsv_operations_sec]
REGEX = "(.*)"\t"(\d+)"\t"(\d+)"\t"(.*)"
FORMAT = Number of consecutive failed operations::$4
</pre>

then in props.conf

<pre>
[tsv]
TRANSFORMS-failcount=windows_perf_tsv_fail
TRANSFORMS-processdelay=windows_perf_tsv_delay
TRANSFORMS-opspersec=windows_perf_tsv_ops_sec
</pre>

However, when I view the data on my linux server (the main host), I can't query by failcount=*. What am I doing wrong? Below