Forums: SplunkGeneral: no data indexed with splunk 2.0

Hi all,

I have made an upgrade to Splunk 2.0 and now I haven't any data indexed. I have made a complete reinstallation : remove splunk 1.x and install splunk 2.

I use syslogFIFO with syslog-ng to feed splunk. Here is my config :

<!--Copyright (C) 2006 Splunk Inc. All Rights Reserved. Version 1.2 -->

<pipeline name="syslogFIFOinput" type="startup">
<processor name="fifoReader" plugin="fifoinputprocessor">
<config>
<field>_raw</field>
<!-- "fifo" to read -->
<fifo>/var/log/splunk.pipe</fifo>
<sourceType>syslog</sourceType>
<host>linprd03.lanprd.nbb</host>
<index>default</index>
</config>
</processor>

<processor name="sendOut" plugin="queueoutputprocessor">
<config>
<queueName>parsingQueue</queueName>
</config>
</processor>
</pipeline>

If I stop splunk and use piper.pl, I see data in /var/log/splunk.pipe, so syslog-ng is ok.

When I go to the web interface of Splunk I receive main : 0 data indexed.

When I go to data, I have for FIFO :

Fifo (1):
name
/var/log/splunk.pipe

Where is the problem ?

Thanks for any help.
Kind regards.
Pierre.

Hi all,

I have found an error msg in the splunk log :

ERROR DiskUsageMonitor - The disk is full stopping information from going to indexer.
06-02-2006 11:47:05.913 ERROR DiskUsageMonitor - Indexing will resume when free disk space rises above : 10000

I have specified during installation :

Indexing : yes | Index Location : /data/splunk/var/lib/splunk
HostName : linprd03.lanprd.nbb
Disk Space Reserve (GB) : 10

Filesystem Size Used Avail Use% Mounted on
/dev/rd/c0d0p2 4.0G 2.0G 1.8G 54% /
/dev/rd/c0d0p1 145M 21M 117M 15% /boot
none 1.9G 0 1.9G 0% /dev/shm
/dev/rd/c0d0p6 2.0G 76M 1.8G 4% /home
/dev/rd/c0d0p5 4.0G 298M 3.5G 8% /var
/dev/rd/c0d1p1 19G 134M 18G 1% /data
/dev/rd/c0d1p2 16G 896M 14G 7% /logs

The data has 18G free, so there is a problem with this param.

Thanks.
Pierre.

Have tried to change to 5 G, but the problem is still there. How can I solve it ??

06-02-2006 15:43:18.050 ERROR DiskUsageMonitor - The disk is full stopping information from going to indexer.
06-02-2006 15:43:18.050 ERROR DiskUsageMonitor - Indexing will resume when free disk space rises above : 5000

Thanks.
Pierre.

Pierre,

I know what's going on here. I had the same problem.

You set your data directory for splunk to a different partition during the install. Unfortunately the installer completely ignores that and puts it's data in the default location of /opt/splunk/var/lib/splunk. So what's happening is that the diskusage processor is doing it's job and trying to keep 5 GB free in your / partition. Since your / partition only has 2 GB free, splunk refuses to write to the disk.

My workaround was to make a symlink from /opt/splunk/var/lib/splunk to /var/lib/splunk [where i TOLD it to put the data..] and set the diskusage processor to a very low number like 10 [megabytes]. I'll document all this in my splunk FAQ when I get it updated for 2.0. In the meantime, email me if you need any help getting yours set up.

By the way, this is a known problem. It's supposed to be getting fixed soon.

cheers,
Joe

Hi Joe,

I have made a link to /data/splunk with this command but :

ln -s /data/splunk /opt/splunk/var/lib/splunk

[root@linprd03 splunk]# cd /opt/splunk/var/lib/
[root@linprd03 lib]# ls -l
total 4
drwxrwxr-x 10 splunk splunk 4096 Jun 6 11:31 oldsplunk
lrwxrwxrwx 1 root root 12 Jun 6 11:35 splunk -> /data/splunk

But, it does not work !!. If I start Splunk, I receive

06-06-2006 13:24:19.725 ERROR DiskUsageMonitor - The disk is full stopping information from going to indexer.
06-06-2006 13:24:19.725 ERROR DiskUsageMonitor - Indexing will resume when free disk space rises above : 5000

If I change <minFreeSpace>1000</minFreeSpace> to 1 G, Splunk starts but don't register any data !.

So,I have a problem with my link.

Can you please tell me how you have defined your link.

Thanks a lot for your help.
Kind regards.
Pierre.

If I set the diskusage to 1G, Splunk starts but I have error messages :

06-06-2006 15:25:43.474 WARN MetaData - Cannot load aggregate meta data from db
06-06-2006 15:25:43.918 WARN MetaData - Cannot load aggregate meta data from db
06-06-2006 15:25:44.130 WARN MetaData - Cannot load aggregate meta data from db
06-06-2006 15:25:44.324 WARN MetaData - Cannot load aggregate meta data from db
06-06-2006 15:25:44.524 WARN MetaData - Cannot load aggregate meta data from db

Thanks.
Pierre.

What I have done is the following :

Install Splunk with the default data path /opt/splunk/var/lib/splunk

Move /opt/splunk/var/link/splunk to /data

Create a link with ln -s /data/splunk /opt/splunk/var/lib/splunk

[root@linprd03 myinstall]# cd /opt/splunk/var/lib/
[root@linprd03 lib]# ls -l
total 0
lrwxrwxrwx 1 root root 12 Jun 6 15:20 splunk -> /data/splunk

Thanks.
Pierre.

Hey Pierre,

I responded to your email this morning. That should get you straightened out. For everyone else's benefit, I believe that the problem is that the data has to be moved or copied from the default directory of $SPLUNK_HOME/var/lib/splunk to the directory that you symlink to. Otherwise splunk will throw those metadata errors and refuse to start properly.

Pierre, please let us know if that got your problem fixed.

Thanks,
Joe

Just a word of warning, unrelated, if you place the /opt/splunk/var/lib/splunk/directorymonitor and /opt/splunk/var/spool/splunk in different partitions you will run into some problems with the directory monitor (batchfile module). If your only using the tailing or fifo stuff you'll be fine but if you want to use the dirmon make sure those two items are on the same partition.

cheers,
Rory

There an even better work around than what Joe has suggested: navigate to your $SPLUNK_HOME/bin directory. Look for the variable SPLUNK_DB and change this to the full path of where you want your index on the partition. The bug here is with the installer, not with the product itself.