Splunk Scripted Input Secrets

Splunk’s Universal Forward has the neat capability of executing arbitrary scripts while capturing their output and sending that to Splunk. This feature allows you to turn any executable, batch file or PowerShell script into a Splunk data source, making the data collection options basically limitless. This post explains a few tricks that are difficult to find otherwise.

Pillar of Darkness Expedition: 1913 by davidd under CC

Scripted Input without a .path File

Scripted input examples often show the convoluted method of using an additional .path file to run the file/script. However, a .path file is only required if the executable is not part of your app. If it is, put it in your app’s bin subdirectory and reference it directly, like so:

[script://.\bin\hksm.cmd]

This needs to go into the inputs.conf file, by the way.

Run Only Once

To run a scripted input only when the Universal Forwarder starts, use -1 as interval, e.g.:

[script://.\bin\hksm.cmd]
interval = -1

Log to Splunkd.log

By default, anything a scripted input prints to stdout (i.e. the console) is captured by the Universal Forwarder and sent to Splunk for indexing, while anything printed to stderr is sent to Splunk’s splunkd.log. That makes it easy to log status messages, e.g. from a Windows batch file:

echo No scripts to process >&2

The only downside is that such messages are treated as error messages and are marked as such in splunkd.log. That is OK if you are logged error conditions, but if you would just like to send the current status for informational purposes, the severity INFO would be much more appropriate. Luckily, that is simple to achieve by prepending the string “INFO” to the messages:

echo INFO No scripts to process >&2

Different Index, Source or Sourcetype

You can send different sourcetypes (or sources/indexes) from one scripted input by prepending the actual message with a “magic” code and the desired index, source and/or sourcetype:

echo ***SPLUNK*** index=indexname source="sourcename" sourcetype="sourcetypename"
echo This is the acual message

This way of dynamically assigning metadata fields is documented here. For it to work, the HEADER_MODE props.conf setting must not be none.

Note: the line containing ***SPLUNK*** above the message is not counted against Splunk’s data volume!

Multiline Events

To send multiline events and stop Splunk from breaking a message up into individual events at line breaks add the following in props.conf on your indexers:

[source::hksm]
BREAK_ONLY_BEFORE = ^\*\*\*SPLUNK\*\*\*

Working Example

Most of the techniques described here have been used in my free Splunk app HK Systems Management.