Forums: SplunkGeneral: JSON Formatted Logs

I have an application which outputs in the format

timestamp|{json}

I did this because I am going to write another application for searching and reporting and the easiest way to read back in this data was for it to be JSON formatted.

Then I thought, perhaps I can just use splunk. However despite the documentation stating that it can handle logs/events in any format, JSON does not seem to be catered for at all (except for regex'ing).

Now I can write a simple java application to parse the logs into simple name value pairs for input into splunk using the Java API, looks like 30 minutes effort, but am lazy and would prefer not to.

Have I missed something ? Any ideas ?

I am in a similar position, looking to get JSON formatted data into splunk. The following are 10 redacted events.

Jan 20 04:01:14 ccw1 cc_log: {"uri":"\/example-path\/node\/12109","act":"view","ccl":"4b56f08ac4705","sid":null,"ip":"127.0.0.1"}
Jan 20 04:01:29 ccw1 cc_log: {"uri":"\/example-path\/node\/22323","act":"view","ccl":"4b56f09928ae7","sid":null,"ip":"127.0.0.1"}
Jan 20 04:01:34 ccw1 cc_log: {"uri":"\/example-path\/node\/98899","act":"view","ccl":"4b56f09e1ad7b","sid":null,"ip":"127.0.0.1"}
Jan 20 04:01:43 ccw1 cc_log: {"uri":"\/example-path\/node\/872348","act":"view","ccl":"4b56f0a705ca4","sid":null,"ip":"127.0.0.1"}
Jan 20 04:01:44 ccw1 cc_log: {"uri":"\/example-path\/node\/22112133","act":"view","ccl":"4b56f0a87e588","sid":null,"ip":"127.0.0.1"}
Jan 20 04:01:49 ccw1 cc_log: {"uri":"\/example-path\/node\/2232331","act":"view","ccl":"4b56f0ad06de2","sid":null,"ip":"127.0.0.1"}
Jan 20 04:01:49 ccw1 cc_log: {"uri":"\/example-path\/node\/555","act":"view","ccl":"4b56f0ad071cc","sid":null,"ip":"127.0.0.1"}
Jan 20 04:01:49 ccw1 cc_log: {"uri":"\/example-path\/node\/2","act":"view","ccl":"4b56f0ad47f6a","sid":null,"ip":"127.0.0.1"}
Jan 20 04:01:57 ccw1 cc_log: {"uri":"\/example-path\/node\/register","act":"register","ccl":"4b56f0b5d0796","sid":null,"ip":"127.0.0.1"}
Jan 20 04:02:06 ccw1 cc_log: {"uri":"\/example-path\/node\/login","act":"login","ccl":"4b56f0beaeba9","sid":null,"ip":"127.0.0.1"}

Let's assume that your sourcetype is "json" - this is props.conf:

[json]
SHOULD_LINEMERGE=true
BREAK_ONLY_BEFORE=^\d+\s+\d{2}\:\d{2}\:\d{2}
TIME_PREFIX=^
TIME_FORMAT=%b %d %T
MAX_TIMESTAMP_LOOKAHEAD=15
REPORT-json=json_kv

This is transforms.conf:

[json_kv]
REGEX = \"([^\"]+)\"\:\"([^\"]+)\"
FORMAT= $1::$2
MV_ADD=true

Thanks araitz it worked great for me.

You may need to add "REPEAT_MATCH=true" to the transform to get all the key value pairs out of the log