Known issues

Known issues

The following are issues and workarounds for version 4.0.6 of Splunk.

Events dated 2010 not returned by searches

Splunk is not auto-recognizing some timestamps from the year 2010. The problem is specific to two-digit year representations; the timestamp for these events are not correctly indexed by Splunk and so the events are not returned correctly by search. This is a particular issue with Windows Event Log events, but affects all events with timestamps that use two digits to represent the year.

If events from 2010 are not returned by searches, replace the datetime.xml file in your Splunk installation with this one:

http://download.splunk.com/support/config/2010fixed.datetime.xml.gz

The datetime.xml file is located in $SPLUNK_HOME/etc. You must apply this file to all indexers, and to regular Splunk forwarders (but you do not have to apply it to light forwarders, since indexing is not occurring on them.)

To apply this file to your instance:

1. Download the file.
2. Decompress it: gzip -d 2010fixed.datetime.xml.gz
3. copy it to your install: cp 2010fixed.datetime.xml path/to/splunk/etc/datetime.xml

This issue will be resolved in 4.0.8. We are currently working on a step-by-step procedure for recovering events between 01/Jan/2010 00:00:00 and the time you replace datetime.xml, but the general recommendation is to examine your buckets, locate those that include events for the timeframe in question, export them, and re-import them. For more information on buckets and how to identify their timeranges, refer to this topic on the Splunk Wiki.

Security

• A cross-site scripting vulnerability has been identified in Splunk Web. To resolve the issue, download this file and untar it into your $SPLUNK_HOME directory. (SPL-27560)

General issues

• When you save a top or rare search with the argument showperc, the showperc argument disappears when you run the search. (SPL-27694)
• On shutdown, many WARN lines are displayed in splunk.log that should actually be INFO. These lines can be safely ignored. (SPL-24862)
• You must manually distribute certificates to a host before you can successfully add it as a distributed search peer using the CLI. (SPL-24786)
• If you expand the view of a large event to the full event and back again to the summary view, subsequent attempts to expand to view the entire event will be restricted to 500 lines. (SPL-27109)
• Show source will show no text if there are more than 10,000 events per second from that host, source and sourcetype. (SPL-26792)
• Admin Activity dashboard reports IE8 as Mozilla. (SPL-27296)
• Saving an event type via Splunk Web adds an extra literal "search" to the full search. (SPL-27049)
• Upgraded versions of Splunk (3.4.x -> 4.x) do not support syslog out. (SPL-27621)
• Upgraded versions of Splunk light forwarders (3.4.x -> 4.x) index locally despite being configured not to. (SPL-26752)
• web_access.log and web_service.log grow forever, and consume unbounded disk space. (SPL-27588)
• Solaris light forwarders configured for auto load balancing and SSL forwarding may show extremely high memory usage in certain situations. (SPL-27702)
• Summary indexing does not work if var/run/splunk and var/spool/splunk are on different filesystems. (SPL-26631)
• The SplunkLightForwarder app *requires* an outputs.conf-style choice of server to forward to. If SplunkLightForwarder is configured on while no target server to transmit data to is specified, the Splunk instance will not forward the data, nor block, it will null-route the dataflow. (SPL-27747)
• splunk-search may crash (in factorCommonTerms) when processing moderately complex boolean compound expressions, especially those involving tags. (SPL-27495)

Data input issues

• monitor inputs using the followTail setting sometimes will index some older events or all events from log files which are updated when not intended. (SPL-23555)
• The no_appending_timestamp = true setting doesn't work in UDP inputs (SPL-26783)
• Splunk generates misleading warnings for plain text logfiles, eg Using charset UTF-8 for events from '....', as the monitor is believed over the raw text which may be ASCII. ASCII is a proper subset of UTF8 so this is a non-problem. (SPL-27498)
• When configuring a TCP input the host= setting is not respected. If you want the host value to be something other than the sending server you will need to modify the host value through props.conf and transform.conf. (SPL-27735)
• A SSL bug can cause light-weight forwarders to consume lots of memory without bounds. To workaround, just disable SSL (SPL-27702)
• A very active monitored file may not be picked up again after rolling. The workaround is to force releasing the fd at EOF with time_before_close = -1 for [inputproc] in limits.conf (SPL-28036)
• A windows evenlog input for an undefined windows event log category will gather the data available in the System eventlog category. (SPL-22613)

Localization, internationalization, and character set issues

• Certain Japanese language OSes, including most versions of Windows, use the ¥ (Yen) symbol to denote backslashes in path names. This can cause issues when monitoring or spooling files, and may require custom regex configurations where a file path is part of the dataset. (SPL-23307)

(This issue is also present in the Japanese PDFs of the documentation.)

App and App development issues

• An issue exists in the first time run experience around input collisions: if you enable the *Nix App, the inputs it adds put their data in the "os" index, which by default is only searchable from the *Nix App interface. If you then try to add /var/log as an input (through the Getting Started App or any other App), an error is displayed stating that this input already exists. (SPL-25138)
• It's possible to get to the setup page for an App without enabling it first. (SPL-24852)
• No dashboards are added to the navigation menus for the Windows and *Nix Apps. (SPL-24933)
• It's not possible to delete views from Manager. (SPL-24908)
• Old modules, templates, and other App components are not deleted on upgrade. (SPL-22494)
• The *Nix App is not supported on AIX. (ENH-3001)
• Simple XML searchPostProcess doesn't work with <chart> and <fields>. (SPL-27248)

Search and search App issues

• Saved searches that contain NOT expressions that include quoted terms will not run properly when loaded from the Searches and Reports menu because the quotation marks do not escape correctly. In some cases, removing the quotation marks is sufficient to work around the issue. (SPL-26944)
• Creating an invalid event type does not generate an error. (SPL-25091)
• The All indexed data dashboard count for number of sources stops incrementing at 10,000 sources. (SPL-27300)
• Searching for an extracted field that does not come from raw event data will not work in a distributed environment. To work around this, manually distribute your fields.conf file to be local on the distributed search peer (SPL-26560)
• Some field extractions may not be run due to some search optimization code. This can potentially break views/dashboards that are relying on extracted fields. To workaround, add "| fields *" to your search as the first "|... " component. Eg. search = sourcetype=iis | fields * | stats count by machineID (SPL-27665)
• The xpath operator does not work. (SPL-26985)

Splunk Web and Manager display issues

• The number of users to display per page in Manager > Users does not retain its state if you change it. (SPL-24896)
• Pausing a search job in the job manager does not update the job's displayed status (SPL-24999)
• If you have cookies disabled or if the server and/or client CPU time are not in sync, you will be returned to the login page. Both machines must have the correct time set when cookie timestamp is verified. (SPL-22393)
• If you schedule a search that's a report and have it emailed, the link that is included in the email will link to something that does not have the chart formatting you specified. (SPL-25671)
• If Splunk Web is slow, or if the user clicks rapidly through pages, the following message may be displayed in red: There was an error requesting the job listing. Status "0". Error message: "". The initial text may vary, but the status zero and the blank error text are consistent. This is a non-error which can be safely ignored. (SPL-26826)

Windows-specific issues

• The crawl feature is not applicable on Windows. (SPL-24843)
• The Windows App was enabled by default in its app.conf file in versions 4.0-4.0.2. Starting in version 4.0.3, it is disabled in this file by default. (SPL-25487) Read on for important details:
  • If you're upgrading from 4.0-4.0.2 to 4.0.3 or later, the Windows App will be disabled, even if it was enabled in the version you're upgrading from.
  • If you're doing a fresh installation of 4.0.3 or later, the Windows App is enabled by default via the MSI and if you want to install it in a disabled state, you must specify this using the SPLUNK_APP msiexec command as described in "Install on Windows via the commandline" .
• In props.conf the source stanza is ignored. Use sourcetype as a workaround. (SPL-25898)
• Specifying a non-existent Windows Event Log category in inputs.conf will acquire general Application Log data and assign it the non-existent category (SPL-22613)
• Forcing roll from hot to warm from the command line requires a backslash on the pipe character before the debug command, eg splunk search "\|debug cmd=roll index=your_index". (SPL-27534)
• In 4.0.6 and earlier, running a light forwarder with deployment client enabled, can cause high CPU usage. To avoid this, disable the deployment client. (SPL-26789)
• A perpetual license may report expiration in approximately 1.5 years. (SPL-27005)

Migrating your license

Splunk 4.x does not work with licenses from older releases. When you install Splunk 4.0.2 or later, your existing 3.x license will be moved aside and replaced with a 4.x Enterprise trial license, which you can use while you procure an updated license.

• If you are an current Enterprise customer, check your splunk.com orders page for an updated license.
• If you are running with a 3.x Free or Enterprise trial license, delete the $SPLUNK_HOME/etc/splunk.license file before you start Splunk 4.x. The instance will then pick up the 60-day Enterprise trial license.
• If you see your license expiration date in year 2283, you will notice that days remaining appear to be off. This is due to the Year 2038 problem. Please request a new license key by submitting a support case.

Considerations for users of Splunk 3.4.x

Splunk 4 is a huge stride forward in performance and flexibility, but there are a few interaction changes vs. 3.4.x which upgraders should be aware of, and even some reasons why you might want to wait for a future release before upgrading. Below are some capabilities that have changed with the introduction of Splunk 4:

Live tail

• With Splunk 4's dramatically improved search and indexing speed, along with the ability to provide intermediate search results, you don't really need a separate live event console to see data in near real-time. However, if your use case relies on version 3.4.x's "Live tail" feature, you may want to wait on upgrading to Splunk 4. Future roadmap plans involve re-architecting the live tail functionality to scale across much larger data flows, and across distributed environments. Additionally, look out for improve real-time alerting and dashboard updates down the road as a result of these upcoming architectural changes.

Custom field actions

• Based on customer feedback, we decided to re-architect this feature to improve flexibility and allow for event actions based on multiple fields. Expect this functionality to be reintroduced in a near term 4.x release. If you rely on this functionality, but still want to upgrade, you may want to consider Splunk 4's new "Dynamic field lookups" as an alternative which allows you to map data from external databases and lists into Splunk.

Snapshots

• In Splunk 4, we've improved upon 3.x's ability to take a timeline snapshots of individual searches. Try out Splunk 4's new job manager which allows you to retrieve the entire cached search result, including reports, from existing searches.

Event scrolling

• In Splunk 4, the new page selector allows you to hop between results with greater flexibility, even as a search runs. However, for those who still prefer a scroll bar, expect this capability to be re-introduced as an option in a future 4.x release.

Timeline and timestamp interaction

• In Splunk 4, we improved the timeline to allow users to quickly view any time range within search results, without having to rerun a search. Also try clicking "zoom-in" on the timeline, which now allows you to lock-in a time range, and specify follow on search.
• We're also planning to improve the usability of some related 3.4.x functionality including clicking on timestamps, and double clicking on timeline bars in future versions of 4.x.

Crawl

• Crawl is no longer configurable via the UI, but is still available as a search command. Based on customer feedback, we have decided to re-architect this feature to make it easier and more effective. Expect improved functionality, along with a new user interface to be introduced in a future release.

FIFO inputs

• This input type has been depreciated with Splunk 4, and we do not recommend using it as a best practice due to data loss considerations. Please contact support@splunk.com if you currently rely on this input type for alternative input methods.

RSS Feed alerts

• Splunk 4 now has improved capabilities for creating email alerts based off searches on your data. However, the RSS feed alerting option from 3.x is currently being re-architected based on customer feedback, and will be reintroduced as an option in a future 4.x release.

Deployment

• Splunk 4.0.x Deployment server is not compatible with Splunk Deployment client 3.x.