Documentation: 3.3
|  Print Version Contents
This page last updated: 08/13/08 06:08pm

Configure character set encoding

Splunk allows you to configure character set encoding for your data sources. Splunk has built in character set specifications to support internationalization of your Splunk deployment. Splunk supports 71 languages (including 20 that aren't UTF-8 encoded). You can Retrieve a list of Splunk's valid character encoding specifications using the iconv -l command on most *nix systems.

Splunk attempts to apply UTF-8 encoding to your sources by default. If a source doesn't use UTF-8 encoding or is a non-ASCII file, Splunk will try and convert data from the source to UTF-8 encoding unless you specify a character set to use by setting the CHARSET key in props.conf.

Note: If a source's character set encoding is valid, but some characters from the specification are not valid in the encoding you specify then Splunk escapes the invalid characters as hex values (for example: "\xF3").

Supported character sets

Language Code
Arabic CP1256
Arabic ISO-8859-6
Armenian i ARMSCII-8
Belarus CP1251
Bulgarian ISO-8859-5
Czech ISO-8859-2
Georgian Georgian-Academy
Greek ISO-8859-7
Hebrew ISO-8859-8
Japanese EUC-JP
Japanese SHIFT-JIS
Korean EUC-KR
Russian CP1251
Russian ISO-8859-5
Russian KOI8-R
Slovak CP1250
Slovenian ISO-8859-2
Thai TIS-620
Ukrainian KOI8-U
Vietnamese VISCII

Manually specify a character set

Manually specify a character set to apply to a source by setting the CHARSET key for a source in props.conf.

For example, if you have a source the is in Greek and that uses ISO-8859-7 encoding, set CHARSET=ISO-8859-7 for that source in props.conf.

[source::$SOURCE]
CHARSET=ISO-8859-7

Automatically specify a character set

Splunk can automatically detect languages and proper character sets using its sophisticated character set encoding algorithm.

Configure Splunk automatically detect the proper language and character set encoding for a source by setting CHARSET=AUTO for that source in props.conf. For example, if you want Splunk to automatically detect character set encoding for the source "my-foreign-docs", then set CHARSET=AUTO for that source in props.conf.

[my-foreign-docs]
CHARSET=AUTO

If Splunk doesn't recognize a character set

Train Splunk to recognize a character set if you want to use an encoding that Splunk doesn't recognize, by adding a character set sample file to the following directory: 

SPLUNK_HOME/etc/ngram-models/_<language>-<encoding>.txt

Once you add the character set specification, then restart Splunk. After you restart, Splunk can recognize sources that use the new character set, and will automatically convert them to UTF-8 format at index time.

For example, if you want to use the "vulcan-ISO-12345" character set, copy the specification file to the following path: 

/SPLUNK_HOME/etc/ngram-models/_vulcan-ISO-12345.txt

Previous: Mask sensitive data in an event    |    Next: Dynamic metadata assignment

Comments

No comments have been submitted.

Log in to comment.