Turning Splunk into a Systems Management Tool

Despite its great power, Splunk is relatively static with regards to the data it processes. You cannot instruct it to simply run a script on all endpoints and index the results. The app HK Systems Management changes that. It turns Splunk into a kind of PsExec on steroids.

Background

Splunk is a fabulous tool for analysing all kinds of data, and there are many different ways of getting data into Splunk. But none of those are even close to being interactive, at least not if you want to collect data from thousands of machines.

Also, being a data analytics tools, Splunk data flow is one way only: from the endpoint to Splunk. There is no way back; at least, normally there is not.

Remote Code Execution With Captured Output

It turns out that morphing Splunk from a tool that collects data on endpoints into a tool that manages endpoints and collects data is not that difficult. Only a few simple configuration changes are required for Splunk to:

• Execute arbitrary command jobs on remote systems (this part is similar to PsExec)
• Capture each job’s output
• Centrally index the output from all jobs
• Provide easy access to each job’s results
• Display job status and output on a dashboard

How It Works

We can send files to endpoints with Splunk’s Deployment Server. The only requirement is that Splunk’s Universal Forwarder is installed on the endpoints. Since Universal Forwarder typically runs as Local System (or similar, depending on the platform) and Deployment Server is able to push down new or changed apps to all forwarders, we can set up a centrally managed app that executes any scripts it gets as a payload with admin privileges on the endpoints.

With a little logic in the app we can even provide a simple way to configure whether scripts run regularly or only once, e.g. by looking for the suffix -runonce in the script name.

Implementation

The Splunk app HK Systems Management implements these concepts. It should be seen as a proof of concept and is in no way comparable to established systems management products. But it proves the point that systems management is totally doable with Splunk.

HK Systems Management consists of three Splunk apps that serve different purposes:

• HK_SystemsManagement_Endpoint: runs on the endpoints you want to manage; this is the main component
• HK_SystemsManagement_Indexer: indexer app, creates the index for storing the data
• HK_SystemsManagement: dashboard app, visualizes the data stored in Splunk

Endpoint App

The endpoint app’s main script hksm.cmd is run by Universal Forwarder as a scripted input every 30 minutes. It does the following:

• It runs all scripts in the UserScripts subdirectory
• If a user script’s name ends with -runonce successful execution is stored in the system profile’s %LocalAppData% directory and the script is not run again
• All output is sent to the index hksm with source hksm
• The script’s output is sent to the local splunkd.log and also to Splunk with sourcetype hksm:main
• The output of user scripts is sent to Splunk with the name of the user script as sourcetype
• Multiline output from user scripts is captured as a single Splunk event

Deployment

Indexer and Search Head

Deploy HK_SystemsManagement to your search head(s) using Splunk’s UI or any other method of your choosing.

The helper apps (TA and SA in Splunk terminology; I prefer to call them indexer and endpoint apps) are located in the install subdirectory of the main app. Locate $SPLUNK_HOME\etc\apps\HK_SystemsManagement\install\HK_SystemsManagement_Indexer and deploy it to your indexer(s).

Endpoints

Locate $SPLUNK_HOME\etc\apps\HK_SystemsManagement\install\HK_SystemsManagement_Endpoint and deploy it to the endpoints via Splunk’s Deployment Server. You can do so with the following configuration:

Edit $SPLUNK_HOME\etc\system\local\serverclass.conf so that it includes the following lines:

whitelist.0 = *

# Define a serverclass 
[serverClass:windows]
# Deploy only to Windows machines
machineTypesFilter = windows-*
 
# Define which apps to deploy to the serverclass
[serverClass:windows:app:HK_SystemsManagement_Endpoint]
stateOnClient = enabled
restartSplunkd = true

Copy the HK_SystemsManagement_Endpoint directory to $SPLUNK_HOME\etc\deployment-apps.

Run the command:

$SPLUNK_HOME\bin\splunk reload deploy-server

Usage

To execute scripts on endpoints simply copy them to $SPLUNK_HOME\etc\deployment-apps\HK_SystemsManagement_Endpoint\bin\UserScripts and instruct Deployment Server to reload:

$SPLUNK_HOME\bin\splunk reload deploy-server

Remember:

• Scripts must have the extension .cmd
• To run scripts only once per endpoint add the suffix -runonce to the script name, e.g. systeminfo-runonce.cmd
• If the script name does not end with -runonce it is executed every time the management script is invoked (by default every 30 minutes)
• To make a runonce script run again delete the corresponding file in C:\Windows\System32\config\systemprofile\AppData\Local\HK Systems Management\Executed Scripts

Download

You can download HK Systems Management in Splunk’s app directory.