Splunk Base : SplunkAdministration

Recovering from catastrophic crash

svisionguy — Thu, 24 Mar 2011 15:01:44 PDT

So we had a catastrophic crash on our Splunk server and had to restore from back up, but after the restore the Splunkd daemon kept crashing. If we do a new install it works fine, but then we can't access our 23gig of DB's with very important data. Is there a way to import our old DB's so that the data can be recovered? This is on a windows system.

Thanks in advance!

Splunk With Cisco ACS

seefor — Mon, 21 Mar 2011 10:07:51 PDT

Any updates on this?

Intermediate Forwarder Cooked Event Data Filter

ephemeric — Mon, 21 Mar 2011 05:27:56 PDT

Hi,

I have the regex transforms working. We changed one light forwarder to tcpoutput raw and it appears good.

So, can we say that regex transforms AKA filters will not happen on cooked event data as opposed to raw TCP input data?

Another problem though is that raw TCP input data is not processed through the intermediate forwarder. Can this be done? TCP raw input into the parsing queue with applied regex transforms and cooked TCP event data output?

Like so: raw TCP input > parsing queue > props|transforms > cooked TCP output... can someone please identify the exact path taken through the data pipeline including which segments and processors are used?

Thanks.

Intermediate Forwarder Cooked Event Data Filter

ephemeric — Thu, 17 Mar 2011 07:18:31 PDT

Hi,

We have a 4.1.6 light forwarder sending to a 4.1.7 intermediate forwarder to a 4.1.7 indexer in default cooked format.

Try as I might the below filter will not work on the intermediate forwarder.
The default SOURCE_KEY = _raw concerns me.
Will the filter be matched on cooked event data?

Thanks.

props.conf
[source::WinEventLog:Security]
TRANSFORMS-null = setnull

transforms.conf
[setnull]
REGEX = EventCode=565
#REGEX = (?m)^EventCode=(565|566|538)\b
DEST_KEY = queue
FORMAT = nullQueue

Default password fails

rachel — Wed, 16 Mar 2011 12:00:35 PDT

these forums are deprecated. please post further questions over at answers.splunk.com. this seems like a support case, though--file a case via the support portal.

error when triggering script by a saved search

Greg_LeBlanc — Wed, 16 Mar 2011 09:52:37 PDT

Hey,

I am getting pretty much the same error when I try to run a Perl script as well.

Have you figured this out?

Thanks.

Remove host data

mntbighker — Tue, 15 Mar 2011 15:15:42 PDT

Bump

Default password fails

mntbighker — Tue, 15 Mar 2011 15:12:33 PDT

So far still not a peep from Splunk team. I deleted 4.1.7 entirely and installed 4.2 with the same result. No way to log into the web interface.

Windows 2008 Event Descriptions not displayed

hughkelley — Thu, 10 Mar 2011 14:47:04 PST

I'm seeing a slightly different phenomenon in version 4.1.7, build 95063.

I get a Message=NULL for my events, even when I look at the source. Is this possibly related?

I'm even more confused because I had this same configuration working in a lab.

20110310224043.000000
Category=14081
CategoryString=NULL
ComputerName=zzzzzzzzz
EventCode=5136
EventIdentifier=5136
EventType=4
Logfile=Security
RecordNumber=349210678
SourceName=Microsoft-Windows-Security-Auditing
TimeGenerated=20110310224043.198864-000
TimeWritten=20110310224043.198864-000
Type=Audit Success
User=NULL
wmi_type=WinEventLog:Security
Message=NULL

Default password fails

mntbighker — Mon, 07 Mar 2011 18:15:04 PST

I have deleted the whole thing and cleaned up all remnants I could find. After reinstalling I have the exact same issue. This one Linux system only? I submitted a ticket on 2/28 and so far no sign of any activity.

Migrating to different Environment.

rachel — Mon, 07 Mar 2011 16:18:17 PST

i think you got your help in #splunk IRC this morning; please ask any future questions at http://answers.splunk.com; it supersedes this forum.

Migrating to different Environment.

karthikkumar — Mon, 07 Mar 2011 04:56:44 PST

Hi,
I am having splunk in one environment(network) with forwarders and have configured lot of searches, alerts and System Configurations ( in Splunk -> Manager ) .. Now I want another very similar setup on another network? Is it possible to export the configurations of the existing Environment and import in another new environment ?? Can I do that ..

splunk error

lubinski — Sun, 06 Mar 2011 17:07:38 PST

Same problem here..

Unable to add Data Inputs due to error

dprice01 — Thu, 03 Mar 2011 12:49:19 PST

That should read psturge said:
hah, oops.

Unable to add Data Inputs due to error

dprice01 — Thu, 03 Mar 2011 12:47:22 PST

permalink said: "For production environments, use a proper log management product like Honeycomb Technologies to feed into Splunk. It will scale much better, and you full remote management, native file access monitoring, and really cool reports/data mining too."

Isn't that was Splunk is supposed to be, or am I missing something? If honeycomb comes back with the right price, what motivation do I have to keep splunk around?

How can I read a MySQL table as a Splunk input?

nunopratas — Wed, 02 Mar 2011 04:46:37 PST

did you manage to suceed? I am having the same problem?
can you please post how.

Splunk - Cannot Update Permissions - Error

hughkelley — Sun, 27 Feb 2011 12:59:15 PST

I'm seeing this in 4.1.7 too. How did you work around it? Which config file holds this setting?

splunk error

bwenge — Sat, 26 Feb 2011 12:17:24 PST

I have got next message

"received event for unconfigured/disabled index='_audit' with source='source::audittrail' host='host::ieprodweb01' sourcetype='sourcetype::audittrail'.(I got that message on Forwarder server and Receiver server)

I think I have got that message when I have configured Light Forwader and Receiver.

I have installed on my forwader server: splunk 4.1.6 trial version
On Splunk server(Receiver),I have installed splunk 4.1.6 free license.

Can you help me to fix that problem.

Regards,

John

Default password fails

mntbighker — Fri, 25 Feb 2011 17:29:45 PST

I have installed splunk on about 10 Linux boxes in the past week. 2 have been 64bit kernels. The second 64bit one installs and appears to start normally. But the default user:pass refuse to work? Anybody have any idea what might be going on?

[Revised on Fri, 25 Feb 2011 17:59:43 -0800]

I can set up forwarding to my Splunk log server. I can get SSL to work for that connection. But I can't disable the local index or turn on lightforward because it says my authentication fails.

monitor apache access log on my linux splunk server

rachel — Fri, 25 Feb 2011 16:06:46 PST

start here:

http://www.splunk.com/base/Documentation/latest/Admin/WhatSplunkcanmonitor

and then ask future questions at answers.splunk.com :)

Audit index has ldap user info. I would like to see ldap Full Name field

rachel — Fri, 25 Feb 2011 16:03:30 PST

you could probably do this with lookups, search for that in the Splunk documentation.

Audit index has ldap user info. I would like to see ldap Full Name field

brantramey — Fri, 25 Feb 2011 15:10:50 PST

I am searching through the _audit index and I am seeing a lot of nice stuff. The problem I am having is it uses the user id for the audit information. That is fine but I would like to be able to see the Full Name from ldap as well. Is there a way to pull that information from ldap within my query?

monitor apache access log on my linux splunk server

bwenge — Wed, 23 Feb 2011 23:34:41 PST

I have a linux server with apache installed.I will like to monitor its access logs on my splunk serverrunning on linux.How to do it

how do i convert the Enterprise licence to free

mntbighker — Tue, 22 Feb 2011 17:08:15 PST

Command line?

Can't download apps within Splunk

dcaldwell — Tue, 22 Feb 2011 01:49:12 PST

Hi,

Just installed Splunk on Ubuntu Server, 64 bit. Everything looks good but I can't download the apps (http://xxx.xxx.xxx.xxx:8000/en-GB/app/launcher/home). I get "Invalid username or password" error when I enter my www.splunk.com credentials. I've logged out and back in to the Splunk website to double-check they're correct.

Any ideas?

David.

Reading Exchange log files only 15 days or newer?

panderson — Mon, 21 Feb 2011 12:03:58 PST

With limited resources but many Exchange servers, I wish to only index the last 15 days of message tracking logs and new logs as they come in for a few days. However, these live log directories contain 90 days of logs and the splunk server and license cannot handle this amount of input. What is the recommended way to accomplish this?

F5 LTM

kentperrier — Wed, 16 Feb 2011 13:15:45 PST

F5 fully supports sending the log data out via syslog

Installing multiple instances on Centos via RPM

williamhutson — Tue, 15 Feb 2011 14:32:01 PST

Hey,

I'm trying to install 2 instances of Splunk on a linux machine having migrated from a windows machine. This process was successful on the windows machine, but I'm having a little trouble on the linux machine.

I tried following the directions given by

http://www.splunk.com/wiki/Community:Run_multiple_Splunks_on_one_machine

by running the command

rpm -i --prefix=/opt/new_directory splunk_package_name.rpm

but got something in the order of

package splunk already installed

from rpm.

Any help would be greatly appreciated. Thanks.

CRASH: Faulting application splunkd.exe, version 0.0.0.0, faulting module msvcr80.dll, version 8.0.5

rvenkatesh — Fri, 11 Feb 2011 17:13:35 PST

We have tried the above steps of stopping splunkdaemon services ( infact uninstalled splunk and re-installed), after a particular time it starts showing these errors

Do we have a fix for this yet?

Thanks,

reached the maximum license violations for this time period

gkanapathy — Thu, 10 Feb 2011 16:01:01 PST

http://answers.splunk.com/questions/322/what-happens-when-i-exceed-my-licensed-limit

http://www.splunk.com/base/Documentation/latest/Installation/AboutSplunklicenses#License_violations

reached the maximum license violations for this time period

0l1v1er — Thu, 10 Feb 2011 13:22:57 PST

Hello,

I had to put a couple of servers in "trivial" mode (verbose++) and this pushed the amount of logs higher than the 500MB/day limit for the free license.

I not have a "reached the maximum license violations for this time period" blue banner. Any idea about how long "this time period" is? It's now been 10 days since the last violation, I'm now back to 113MB/day.
What does the "Violation Period: 30" below means? I suppose it's 30 days, but of what? you can have 3 violations in 30 days? you will be locked out for 30 days?

{{

/opt/splunk/bin/splunk show license

Current Daily Usage Amount: 113254140
Expiration date: 2037-01-20T14:30:11-0500
Expiration State:
License level: 500 MB
Product: free
License violations:
2011-02-01T00:04:32-0500 License violation #4
2011-01-31T00:00:35-0500 License violation #3
2011-01-30T00:04:40-0500 License violation #2
2011-01-29T00:00:15-0500 License violation #1
Max Violations: 3
Peak usage: 1103 MB
Days remaining: 9475 day(s)
Violation Period: 30
}}

Thanks

Olivier

LightWeightForwarder stops sending data

rachel — Tue, 08 Feb 2011 14:23:32 PST

i recommend you guys post your questions to answers.splunk.com; it supersedes this forum.

LightWeightForwarder stops sending data

AppServices — Tue, 08 Feb 2011 13:12:58 PST

Have you been able to find a solution to this issue? I'm experiencing the same problem.
Additionally what I've found is when I run "netstat -an|grep <port>" on the forwarding server there is usually a few connections in a FIN_WAIT state.
I'm trying to track this down on the server side.

Thank you

More SNMP Windows frustration..

cbdick — Sat, 05 Feb 2011 15:29:13 PST

It would be nice if you shared what Splunk shares with you. I am about to undertake a similar task ie: getting Splunk that is installed on Windows Server 2008 64-bit to catch SNMP traps from various devices and was hoping this thread will provide me some answers.

Indexing a CSV data file with more than one set of information

rachel — Wed, 02 Feb 2011 10:47:58 PST

hi, i recommend you ask this over at answers.splunk.com -- it is much more active and supersedes this forum.

More SNMP Windows frustration..

kristiaan_d — Tue, 01 Feb 2011 08:02:39 PST

never mind, splunk are contacting me about this issue and how to get splunk for windows setup to capture data correctly.

feel free to close / delete this post.

More SNMP Windows frustration..

kristiaan_d — Tue, 01 Feb 2011 06:41:46 PST

Hi everyone, i need some help as i am slowly starting to go off splunk and its logging all of this due to SNMP...

All i want to-do is setup Splunk so it will index all the SNMP data that my firewall pumps out, (its a watchguard firebox). i have been looking into this and so far im completely baffled as to why this seems to difficult.

i have failed completly to setup NET-SNMP or SNMPTrapd or what ever its called, the help files and manuals are not setup in any way to help out someone like me who has never used the software.

Splunks website states i should ammend a config file

C:\usr\etc\snmp\snmptrapd.conf

this file does not exist in my setup, i have an SNMP.conf file...

all i want to-do is setup splunk to capture the data from my firewall, why is this being made so difficult? i would have thought that a system developed to capture and index data would have the ability to capture SNMP traffic natively?

This appears to be an oversight of someone in project planning, there decision rather to farm it off on some half baked un-usable linux conversion that is no use to someone with little linux config experience.

dont get me wrong if the net-snmp project had a nice write up on how to configure it for windows and to test its working i would not be writing this now but splunk chose to suggest them...

on a final rant, are we ever likely to see someone at splunk write some native SNMP handling into it? might be an idea been as SNMP is encorporated into 99% of devices splunk is designed to index the data from?

kris

Indexing a CSV data file with more than one set of information

phoenixdigital — Sun, 30 Jan 2011 19:12:32 PST

Hi All,

Just curious about the best method to index a CSV file with multiple sets of data inside?

The basic format of the whole file is

I,DataSet1_FieldName1,DataSet1_FieldName2,DataSet1_FieldName3
D,this,54,fred
D,this,87,barry
I,DataSet2_FieldName1,DataSet2_FieldName2,DataSet2_FieldName3
D,784,moreInfo,thatData
D,5443,moreInfo2,thisData
D,524,moreInfo2,theOtherData
I,DataSet3_FieldName1,DataSet3_FieldName2
D,Wow,SoMuchData
D,Really,MoreData

and on and on it goes with about 5 sets of data.

I have been playing around with regular expressions trying to catch one particular set of data but have had no luck yet.

Can someone point me in the right direction as to how I would index data files such as this?

[Revised on Sun, 30 Jan 2011 19:13:52 -0800]

Another note as you can see the first field of each row determines if the row is a header/index row or a data row
I = Index/Header row
D = Data row

JMX/JMS

jordancrombie — Wed, 19 Jan 2011 15:04:47 PST

JMX/JMS would rule...

unable to share dashboard/view

mdurkin — Fri, 14 Jan 2011 11:00:10 PST

I was able to find the answer here:

http://answers.splunk.com/questions/5309/can-a-regular-user-promote-views-dashboards-to-be-app-level-visible

unable to share dashboard/view

mdurkin — Fri, 14 Jan 2011 10:18:09 PST

Hi,

I have a view that belongs to the search app that I am trying to share with other users. I checked and my user does have write permission to the search app. When I try to save the new permission settings, I get hte following message:

Splunk could not update permissions for resource data/ui/views Client is not authorized to perform requested action

I look in web_service.log and see the following error messages:

2011-01-14 13:16:07,358 ERROR [4d3092e756a473eac] admin:1334 - Splunk could not update ACL with the following params: {'owner': u'testuser', 'sharing': u'app', 'perms.read': u'*'}
2011-01-14 13:16:07,446 INFO [4d3092e756a473eac] cached:69 - memoized decorator used on function <function getEntities at 0x8a121b4> with non hashable arguments

If it matters I am running Splunk 4.1.2.

-Mike

LightWeightForwarder stops sending data

arapozo — Tue, 11 Jan 2011 06:57:13 PST

I have a Indexer that's receiving data from some servers but one of them, it stops sending data after a while.
To see if it was a bug or something I updated the splunk version to the latest one, and it started sending but then it stopped and even if I restart splunk it doesn't keep sending. I wen through the splunkd.log and there are some events that keep repeating over and over again and I think the problem lies here:

01-05-2011 19:17:03.103 WARN TcpOutputProc - TcpSendThread: Connection to server XXX.XXX.XXX.XXX:9995, fd:1264 lost - retrying: winsock error 0 01-05-2011 19:17:03.103 INFO TcpOutputProc - attempting to connect to XXX.XXX.XXX.XXX:9995... 01-05-2011 19:17:03.119 INFO TcpOutputProc - Connected to XXX.XXX.XXX.XXX:9995 01-05-2011 19:17:03.119 WARN TcpOutputProc - TcpSendThread: Connection to server XXX.XXX.XXX.XXX:9995, fd:1328 lost - retrying: winsock error 0 01-05-2011 19:17:03.119 INFO TcpOutputProc - attempting to connect to XXX.XXX.XXX.XXX:9995... 01-05-2011 19:17:03.135 INFO TcpOutputProc - Connected to XXX.XXX.XXX.XXX:9995 01-05-2011 19:17:03.135 WARN TcpOutputProc - TcpSendThread: Connection to server XXX.XXX.XXX.XXX:9995, fd:1256 lost - retrying: winsock error 0 01-05-2011 19:17:33.137 INFO TcpOutputProc - attempting to connect to XXX.XXX.XXX.XXX:9995... 01-05-2011 19:17:33.152 INFO TcpOutputProc - Connected to XXX.XXX.XXX.XXX:9995

Splunk is complaining about a missing index...that is there

rachel — Mon, 10 Jan 2011 12:54:25 PST

hi, i recommend posting this to answers.splunk.com if you haven't already. this forum is superseded by answers.splunk.com.

Splunk "one-way-drop" that can receive data, but cannot be compromised by a hacker through some acci

hans135 — Sat, 08 Jan 2011 00:58:01 PST

Hello

Would Splunk work if it's installed on a Linux system based upon UDP so one-way datagrams can be used forward to a device that cannot respond?

Any feedback is appreciated very much. Thank you!

John

Some informations about that topic:

Under 3.6 of Robert Grahams sniffer, e..g. found under http://newdata.box.sk/2001/jan/sniffing-faq.htm

-> Why would I want to do this?
-> For security reasons. Networking is full of accidents waiting to happen, which
-> crackers/hackers exploit in order to break into systems. Clipping the transmit
-> wires on an Ethernet adapters generates a "one-way-drop" that can receive
-> data, but cannot be compromised by a hacker through some accident.

-> Examples:
-> * Receiving syslog messages and storing them to a non-compromisable
-> system. The 'syslog' protocol is used by numerous UNIX services to log security
-> events, and is based upon UDP so one-way datagrams can be used forward to
-> a device that cannot respond. ARP and route tables need to be manually
-> configured to ensure this operation.

-> * Similarly receiving SNMP traps, which also use UDP. Many systems generate
-> SNMP traps in response to security related events.

Splunk is complaining about a missing index...that is there

groupon — Tue, 04 Jan 2011 12:41:16 PST

I'm running v 4.1.6 and getting a similar error:
ERROR IndexProcessor - received event for unconfigured/disabled index='_audit' with source='source::audittrail' host='host::tm22-s00261' sourcetype='sourcetype::audittrail'

I'm guessing that we've nuked something in one of the config files (we deploy Splunk via chef and write our own config file) but I don't know what's missing.

Server additions problem

xiaochuxi929 — Sun, 26 Dec 2010 19:13:22 PST

æ æ³åå¾æ°æ®ï¼In handler 'win-wmi-enum-eventlogs': External handler failed with code '1' and output: Traceback (most recent call last): File "D:\Program Files\Splunk\bin\runScript.py", line 69, in <module> execfile(REAL_SCRIPT_NAME) File "D:\Program Files\Splunk\etc\system\bin\wmi_enum_eventlogs.py", line 28, in <module> admin.init(WMIFindEventLog, admin.CONTEXT_NONE) File "D:\Program Files\Splunk\Python-2.6\Lib\site-packages\splunk\admin.py", line 79, in init msgNode.text = str(exMsg) File "lxml.etree.pyx", line 821, in lxml.etree._Element.text.set (src/lxml/lxml.etree.c:32905) File "apihelpers.pxi", line 650, in lxml.etree._setNodeText (src/lxml/lxml.etree.c:15144) File "apihelpers.pxi", line 1247, in lxml.etree._utf8 (src/lxml/lxml.etree.c:19727) ValueError: All strings must be XML compatible: Unicode or ASCII, no NULL bytes .

Splunk seems to be ignoring props.conf and transforms.conf

dps — Sat, 18 Dec 2010 15:35:48 PST

Which app are you putting this in? Try putting this in the search app.

Dan

Splunk - Cannot Update Permissions - Error

scongdon — Tue, 14 Dec 2010 06:04:39 PST

Splunk support was nice enough to look into this for me...
So, Just FYI if anyone else has noticed this.

After further review, it looks like the problem is that UI-level permissions should not apply to data inputs, and that the manager page for the âevent log collectionsâ is improperly built in 4.1.6. The status column should definitely only show the âenableâ and âdisableâ actions instead of UI-permissions.

Windows 2008 Event Descriptions not displayed

jkanaszka — Mon, 13 Dec 2010 14:33:23 PST

same issue with 4.1.6....any resolutions?

splunkweb fails to start

rachel — Tue, 30 Nov 2010 17:28:40 PST

hey there! i recommend you ask your questions over at answers.splunk.com, it's much more active than these forums.