Splunk Base : SplunkGeneral : #3864

JSON Formatted Logs

davidsg — Tue, 09 Mar 2010 09:34:11 PST

Thanks araitz it worked great for me.

You may need to add "REPEAT_MATCH=true" to the transform to get all the key value pairs out of the log

JSON Formatted Logs

araitz — Tue, 02 Feb 2010 15:48:36 PST

Let's assume that your sourcetype is "json" - this is props.conf:


[json]
SHOULD_LINEMERGE=true
BREAK_ONLY_BEFORE=^\d+\s+\d{2}\:\d{2}\:\d{2}
TIME_PREFIX=^
TIME_FORMAT=%b %d %T
MAX_TIMESTAMP_LOOKAHEAD=15
REPORT-json=json_kv

This is transforms.conf:


[json_kv]
REGEX = \"([^\"]+)\"\:\"([^\"]+)\"
FORMAT= $1::$2
MV_ADD=true

JSON Formatted Logs

rustaml — Tue, 02 Feb 2010 14:51:33 PST

I am in a similar position, looking to get JSON formatted data into splunk. The following are 10 redacted events.

Jan 20 04:01:14 ccw1 cc_log: {"uri":"\/example-path\/node\/12109","act":"view","ccl":"4b56f08ac4705","sid":null,"ip":"127.0.0.1"} Jan 20 04:01:29 ccw1 cc_log: {"uri":"\/example-path\/node\/22323","act":"view","ccl":"4b56f09928ae7","sid":null,"ip":"127.0.0.1"} Jan 20 04:01:34 ccw1 cc_log: {"uri":"\/example-path\/node\/98899","act":"view","ccl":"4b56f09e1ad7b","sid":null,"ip":"127.0.0.1"} Jan 20 04:01:43 ccw1 cc_log: {"uri":"\/example-path\/node\/872348","act":"view","ccl":"4b56f0a705ca4","sid":null,"ip":"127.0.0.1"} Jan 20 04:01:44 ccw1 cc_log: {"uri":"\/example-path\/node\/22112133","act":"view","ccl":"4b56f0a87e588","sid":null,"ip":"127.0.0.1"} Jan 20 04:01:49 ccw1 cc_log: {"uri":"\/example-path\/node\/2232331","act":"view","ccl":"4b56f0ad06de2","sid":null,"ip":"127.0.0.1"} Jan 20 04:01:49 ccw1 cc_log: {"uri":"\/example-path\/node\/555","act":"view","ccl":"4b56f0ad071cc","sid":null,"ip":"127.0.0.1"} Jan 20 04:01:49 ccw1 cc_log: {"uri":"\/example-path\/node\/2","act":"view","ccl":"4b56f0ad47f6a","sid":null,"ip":"127.0.0.1"} Jan 20 04:01:57 ccw1 cc_log: {"uri":"\/example-path\/node\/register","act":"register","ccl":"4b56f0b5d0796","sid":null,"ip":"127.0.0.1"} Jan 20 04:02:06 ccw1 cc_log: {"uri":"\/example-path\/node\/login","act":"login","ccl":"4b56f0beaeba9","sid":null,"ip":"127.0.0.1"}

JSON Formatted Logs

araitz — Mon, 18 Jan 2010 07:38:52 PST

Can you post a sample event or two?

JSON Formatted Logs

lordbuddha — Sat, 16 Jan 2010 19:39:53 PST

I have an application which outputs in the format

timestamp|{json}

I did this because I am going to write another application for searching and reporting and the easiest way to read back in this data was for it to be JSON formatted.

Then I thought, perhaps I can just use splunk. However despite the documentation stating that it can handle logs/events in any format, JSON does not seem to be catered for at all (except for regex'ing).

Now I can write a simple java application to parse the logs into simple name value pairs for input into splunk using the Java API, looks like 30 minutes effort, but am lazy and would prefer not to.

Have I missed something ? Any ideas ?