Splunk Scripted Input Secrets
Splunk’s Universal Forward has the neat capability of executing arbitrary scripts while capturing their output and sending that to Splunk. This feature allows you to turn any executable, batch file or PowerShell script into a Splunk data source, making the data collection options basically limitless. This post explains a few tricks that are difficult to find otherwise.
Scripted Input without a .path File
Scripted input examples often show the convoluted method of using an additional .path file to run the file/script. However, a .path file is only required if the executable is not part of your app. If it is, put it in your app’s bin subdirectory and reference it directly, like so:
This needs to go into the inputs.conf file, by the way.
Run Only Once
To run a scripted input only when the Universal Forwarder starts, use -1 as interval, e.g.:
[script://.\bin\hksm.cmd] interval = -1
Log to Splunkd.log
By default, anything a scripted input prints to stdout (i.e. the console) is captured by the Universal Forwarder and sent to Splunk for indexing, while anything printed to stderr is sent to Splunk’s splunkd.log. That makes it easy to log status messages, e.g. from a Windows batch file:
echo No scripts to process >&2
The only downside is that such messages are treated as error messages and are marked as such in splunkd.log. That is OK if you are logged error conditions, but if you would just like to send the current status for informational purposes, the severity INFO would be much more appropriate. Luckily, that is simple to achieve by prepending the string “INFO” to the messages:
echo INFO No scripts to process >&2
Different Index, Source or Sourcetype
You can send different sourcetypes (or sources/indexes) from one scripted input by prepending the actual message with a “magic” code and the desired index, source and/or sourcetype:
echo ***SPLUNK*** index=indexname source="sourcename" sourcetype="sourcetypename" echo This is the acual message
Note: the line containing
***SPLUNK*** above the message is not counted against Splunk’s data volume!
To send multiline events and stop Splunk from breaking a message up into individual events at line breaks add the following in props.conf on your indexers:
[source::hksm] BREAK_ONLY_BEFORE = ^\*\*\*SPLUNK\*\*\*
Most of the techniques described here have been used in my free Splunk app HK Systems Management.