Forums: SplunkAdministration: routing selective events between splunk indexers

Performing a routing POC with two Splunk 4.0.4 instances:
Ubuntu <--- VirtualBox[WinXP]

Trying to follow:
http://www.splunk.com/base/Documentation/4.0.4/Admin/Routedatatodifferentplacesbasedoncontent

to setup routing of a particular sourcetype.

I set up a TCP Data Input on port 9998 of the Ubuntu host.

Then on WinXP host I configured (props, transforms, and outputs) within
SPLUNK_HOME/etc/system/local as:

[WMI:FreeDiskSpace]

TRANSFORMS-routing=ROUTE2NIX-FD

REGEX=.

DEST_KEY=_TCP_ROUTING

FORMAT=winSharedGroup

server = 192.168.1.100:9998

disabled = false

Re-started both servers (several times now).

One odd thing is, although the Data Input (9998) on the Ubuntu host shows up in lsof -itcp:9998

It now longer appears through Splunk Manager >> Data Inputs. It says I have zero
TCP inputs.

On the Windows host I do see an established connection from Win --> Ubuntu:9998

Any thoughts? Steps I'm missing?

You had the idea right, but you got bitten by a bit of Splunk esoterica. The problem is that you can't do any TRANSFORMS (which includes routing) based on the [WMI:FreeDiskSpace] sourcetype, because the data is not actually set to the value until later.

The result is that your rule never matches, and your TRANSFORM-routing never gets applied.

What you have to do instead is:
props.conf:
[wmi]
TRANSFORMS-routing=routing1,routing2,routing3

transforms.conf:
[routing1]
REGEX=[\r\n]+wmi_type=FreeDiskSpace[\r\n]+
DEST_KEY=_TCP_ROUTING
FORMAT=winSharedGroup

[routing2]
REGEX=[\r\n]+wmi_type=OtherWMIType[\r\n]+
DEST_KEY=_TCP_ROUTING
FORMAT=otherRoutingGroup

and so on for any other routings you need to specify. You can use your REGEX to match more than one wmi_type, so perhaps you still only need a single rule in your TRANSFORMS-routing.

Well, with your help I bit back. That worked. Thanks!

However he is still gnawing on me a bit.

What if I wanted to keep the events I'm routing on the server AND route them to an additional server?

I'm exploring this in order to determine how to share certain events between Splunk servers since our organization plans on installing 3 instances of Splunk controlled by three distinct groups within our organization.
Splunk- A) Splunk-B/C indexers will give distrib. search access to Splunk-A.
Splunk-A may route some events to Splunk-C.
Splunk-B) Will route some events to Splunk-C.
Splunk-C) Splunk-C may give distributed search access to Splunk-B

Also, when you said:
"...because the data is not actually set to the value until later."

Does that imply I'm forwarding raw data and not events?

I was under the impression I could forward cooked events.
Based on that idea I'm also assuming cooked events do not count against the receiving Splunks license limit since it's already been indexed by another Splunk server. True/False?

First question, there is a setting in outputs.conf "indexAndForward" that will keep events on the server and forward. However, everything you index must also be forwarded. If you want to be selective, you'll have to do it on the receiving side by discarding events.

No, you are forwarding cooked data. but at the time you are testing the data to be forwarded, the sourcetype has not been set yet.

Cooked events do count against the license limit. They may have been indexed, but you're indexing them in a different server.

Oh,
With cooked events counting against the limit on both servers that changes things.
We don't want that.

Can we use distributed search to give some Splunk indexer+web servers complete access to certain Splunk indexer+web servers
AND
Give other Splunk indexer+web servers limited search to certain Splunk indexer+web servers?

E.g. Multi-tenant distributed search (where A-B are indexers+web):
Splunk-A) Can search itself and all of Splunk-B/C
Splunk-B) Can search itself and all of Splunk-C
Splunk-C) Can search itself and some events on Splunk-A/B

Forwarding will occur from multiple host to their respective group's Splunk indexer+web server.

If yes, I'm not asking for complete details.
What are the key config points to consider set this up?

Based on looking around forum/docs:
1) Enable distributed search and distribute pub keys to search nodes.
2) Control what can be searched on a search node using:
http://www.splunk.com/support/forum:SplunkAdministration/3403
Does this imply separate indexes are required to give a searcher partial
access to events on foreign node being searched?
3) I also see "Search Restrictions" associated with a user's role. Is the same as
(2) except just setting it through the Web form?
If this is part of it, does this mean user/role info needs to be coordinated across Splunk Indexers?

I am going to try setting this up between two servers.

Update:
I was able to get back to this late in the day.

1) Configured Splunk-1 running on VirtualBox[WinXP] for distributed search to
Splunk-2 running on Ubuntu.

2) Create a directory:
Splunk-2:$SPLUNK_HOME/etc/auth/distServerKeys/ Splunk1_HOSTNAME

Where
Splunk-1_HOSTNAME
was found in Splunk-1 Web >> Manager >> System settings

3) scp'd
Splunk-1:$SPLUNK_HOME/etc/auth/distServerKeys/trusted.pem
to
Splunk-2:$SPLUNK_HOME/etc/auth/distServerKeys/ Splunk1_HOSTNAME

4) Created a new user with a new role where I defined search restriction for this users role: sourcetype=interfaces

Had a "duh" moment where I was able to search for anything on Splunk-2. Then figured out (duh!) that I was logged in as admin instead of the user I just created.

After logging in with the new user I could only get results from Splunk-2 which matched sourcetype "interfaces" -- which was I wanted.

What I find a little odd is that I can only see event results from the time dist. search was activated.

Is that normal?