Forums: SplunkGeneral: Transitive meta events

I've finally managed to create meta events from my log data. However, I haven't quite figured out how to use them. I'm particularly interested in transitive meta events as described in the documentation:

http://www.splunk.com/doc/3.1.4/admin/HowMetaEventsWork
Events can also be linked transitively - if events A and B have a common value, and events B and C have a different common value, then all three can be part of the same meta event.

How would you configure such a meta event? The documentation gives the example of sendmail and exchange logs but in the end, these events all have the same attribute: msgID

All I've managed to do is extract an attribute (for example an ID) by defining the regex in transforms.conf and props.conf as described in the documentation but then Splunk creates one metaevent for each ID. So for example if I have 100 different IDs, I will have 100 metaevents in "index::metaevents".

How do I move from this to transitive meta events? Say A and B share a msgID in common but B and C share a numID?

What's more, it's possible to perform a search on a field used in a metaevent (for example search "msgID=200" but the msgID does not appear as an option in the dropdown list of fields on the searchpage?

I'd be grateful if someone could clarify all this for me.

I've been trying to avoid answering this because (transitive) meta events should be going away soon in favor of trans-am (a transaction processor and algorithm which works at search time). That said, here's what I can tell you:

• To be clear, we are referring to the mathematical definition of "transitively":
  • "Of or relating to a relationship between three elements such that if the relationship holds between the first and second elements and between the second and third elements, it necessarily holds between the first and third elements."

• Thus, in the example given in the docs, the sendmail log has both the qid and the mid in it; thus, when searching for either the qid or the mid, you learn transitively that element 1 is related to element 3 by identity (element 2), the three elements being:
  • The sendmail messages with the qid only;
    • The sendmail messages with the qid and the mid;
      • The exchange messages with the mid;

• To get all drop-down fields that the transitive events have, you will need to extract them to the _meta key as well.

I hope that helps, happy new year.

Thanks for your answer (and of course Happy New Year with a bit of delay)

Are metaevents the current method of "tracking" trades in transactions in Splunk until the transaction processor/algorithm comes out? Or is there another way of doing this that I may have missed out?

If you only have one common field (for example, "tradeid") and it is the same (for example, "123456") across your logs, you do not necessarily need metaevents unless you want all of the events displayed as one or you want to find transitive events. You can do something like:

> tradeid="123456"

to get a bunch of discreet events that contain that trade id.

I kind of get the idea behind the meta events (I think) as in this feature builds a new event (meta-event) out of several events that have a feature in common. There's the example of all the events with the same ip address in the admin documentation. The benefit of this is not obvious to me as a simple search as you described would return the same information.

In my case, I definitely need the transitive meta-event feature as the ids that link the various logs are different.

I was just wondering if there was a preferable way of doing this (while waiting for the transam feature in an official release).

I've been implementing meta events into some of our data. My main concern is that, having a distributed system with a central indexer and several forwarders tailing local log files, the extracting of the metaevents (when data is written to the "cluster index" is done at the forwarders. The trouble is I want my metaevents over data from several forwarders...

Any ideas as to how I can do this?

By the way, I have checked with support: extracting of fields and writing to the index is part of the "cooking" done at the forwarders.

Correct, certain fields are added to the event on the forwarder, but this information is not "indexed" - it is just appended to the data before it is forwarded. The impact is not the same as indexing either.

Since meta events are being deprecated, as I mentioned in the first post, the replacement transaction search ("transam" search command) will handle this at search time, so it won't matter where the events came from.

Won't this be a bit of an issue as all the processing power will be used at search time: for example you might be generating 1 million events every couple of events and you need to search through data from the last hour to find your transaction...

No, processing power at search time is really not a bottleneck. When you use a beefy machine as I recommend, you get plenty of speed and good concurrency. Search time also usually means processing data that is resident in memory.

Have you used transam in preview? What are your impressions of it?

I find it a much more flexible and user-friendly tool than the metaevents.

This might have been specific to my data but metaevents didn't seem to work across multiple hosts in a distributed environment: I guess this would make sense if the metaevents data is written at "cooking" time so transam is a welcome change.

My only problems are:
- milliseconds do matter in my timestamps and as Splunk only appears to take into account the second as the highest level of time granularity, I do end up with events that are a bit mixed up.
- I'm a bit concerned about speed but then my test machine is seriously under-par and am hoping to purchase new machines

in terms of beefiness of the machines: I am not able to run Splunk on a machine with 2 3.4GHz processors but I do have machines with 4 dual-core 2.4Ghz processors and 16 or 32GB of RAM. How would you expect this to perform and compare with your recommended hardware?