Forums: SplunkAdministration: REGEX confusion. Tools say it works, but Splunk isn't buying

I have the following line I am trying to REGEX by pulling the following "sent status msg to all nodes" and dump the line to nullQueue.

2009-09-28 16:44:37.347 [1094797664] >TRACE: clssnmSendingThread: sent status msg to all nodes

In my props.conf file I have the following:
[syslog]
TRANSFORMS-sysl0g=setnull,.........,pruneoraclestatusmsg

and the following in my transforms.conf:

[pruneoraclestatusmsg]
REGEX = \w+\:\s+(sent status msg to all nodes)$
DEST_KEY = queue
FORMAT = nullQueue

I have verified the REGEX line in multiple tools to ensure it is correct, but for some reason Splunk will not dump it to nullQueue and I find it when I search on it.

I have multiple other entries that dump to nullQueue and they work like a champ. Any of the "ftp"entries works. Here is one example:

Props.conf
[syslog]
TRANSFORMS-sysl0g=setnull,pruneftp,pruneftpd,prunevsftpd,prunexinetd,prunevsftpd-pid,pruneoraclestatusmsg

Transforms.conf

1. added to remove xinetd from entering index - PaS, 05212009

[prunexinetd]
REGEX = \<\d+\>(xinetd)\[\d+\]\:
DEST_KEY = queue
FORMAT = nullQueue

Any thoughts on why this doesn't work?

pstein
a part of v4.0...literally.

Do you really need the $ on the end of your regex? It is just an idea but maybe the EOL is different for this sourcetype than your pattern checker.

Even though I doubt that would have any affect, I tried it and the same result. Splunk is still indexing the data. Hmmmmm

Paul,

That's a great "signature" on your first post!

Let's eliminate the easy stuff - how is this data getting to the indexer? Can you verify that the sourcetype is syslog? Did you put the props and transforms on the right instance (forwarder or indexer)?

Like the other nullQueue entries, I put this on the indexer that was seeing the message.
But, you may be onto something that it wasn't coming from syslog, but instead from a Splunk forwarding server. The forwarding server was an Oracle server and using sourcetype: oracle_host(defined by us) to get the data back to the indexing server. Since this is the case, how should I massage the props.conf file on the indexer? Instead of using
[syslog]
TRANSFORMS-sysl0g=pruneoraclestatusmsg

should I be using something like this:

[oracle_host] < -------sourcetype
TRANSFORMS-oracle_host=pruneoraclestatusmsg

So here is what I have setup on my indexing server:

Props.conf
[oracle_host]
TRANSFORMS-oracle_host=pruneoraclestatusmsg

Transforms.conf
[pruneoraclestatusmsg]
REGEX = \w+\:\s+(sent status msg to all nodes)
DEST_KEY = queue
FORMAT = nullQueue

From reading the ../README/props.conf.spec

With the changes above and a restart of Splunk on the indexing server I am still unsuccessful on removing this data before it gets to the index.

Try putting those on the forrwarder and see if that makes a difference - since you are using the heavyweight forwarder the props and transforms will need to be applied there.