Logs & Metrics

This article introduces newbies to writing search queries in LogQL. It should help you get started with building Grafana dashboards based on log data in Loki. It would have helped me immensely building my first Grafana dashboard.

Splunk search results can be exported from the UI as CSV, JSON, and XML, but not as HTML. This article presents a PowerShell script that converts exported search results from JSON into an HTML table for use with documentation, blogs, etc.

When I tried to align the Union Jack and the flag of Germany on a presentation slide I noticed that I couldn’t - their aspect ratios are different. A quick search led me to this list of aspect ratios of national flags on Wikipedia. Apparently, national flags are far from standardized. A broad range of aspect ratios is being used. I decided to start a little fun project finding out which aspect ratios are most common.

When Splunk displays an orange warning triangle instead of a chart or table it is time to investigate. Start by clicking the triangle to bring up a dialog with the error message. In my case that looked like this:

When sizing a new environment it is tempting to use averages. It seems the logical thing to do. But it also guarantees a bad user experience. Example: Sizing an RDS or XenApp Farm Let’s say you’re tasked with building a new Citrix XenApp farm. Being a diligent IT person you set up a pilot: one or two machines with all the right software and settings. Then you carefully select a group of pilot users in such a way that they represent the organization’s employee types statistically correctly. Then you let them work on the new platform, ironing out bugs and such. At the end of that period, you have a great new platform. But there is one big question left: how many servers to buy?!

This article is based on my Splunk .conf 2015 session and is the second in a mini-series on Splunk data model acceleration. Make sure to read parts 1 and 2 first.

This article is based on my Splunk .conf 2015 session and is the second in a mini-series on Splunk data model acceleration. Make sure to read part 1 first. Under the Hood HPAS Population The high-performance analytics store (HPAS) is populated by scheduled searches that run every 5 minutes. The HPAS spans a user-defined time range. Old events are purged automatically by a maintenance process that runs every 30 minutes.

This article is based on my Splunk .conf 2015 session and is the first in a mini-series on Splunk data model acceleration. Why Accelerate? Have you ever seen this? Splunk is great and very fast with needle in a haystack searches, e.g. find a specific keyword in millions of events. It is not so fast with searches that perform calculations on millions of events, e.g. the sum or average of fields.

Splunk’s Universal Forward has the neat capability of executing arbitrary scripts while capturing their output and sending that to Splunk. This feature allows you to turn any executable, batch file or PowerShell script into a Splunk data source, making the data collection options basically limitless. This post explains a few tricks that are difficult to find otherwise.

Despite its great power, Splunk is relatively static with regards to the data it processes. You cannot instruct it to simply run a script on all endpoints and index the results. The app HK Systems Management changes that. It turns Splunk into a kind of PsExec on steroids.

I am more than happy to announce that I won a Splunk’s Revolution Award in the category developers. The winners were presented on the big screen during the keynote. My slide reads:

You have probably heard of Splunk, but can you describe what it does to a colleague in a few sentences? That is not easy. Splunk does not belong in any traditional category but stands apart from the crowd. That makes it interesting, but also the explaining harder. Here is my attempt.

This is a collection of useful tips and resources for developing Splunk apps. For an explanation of the available app types please see my earlier article on the topic. Splunk Search References and Guides Splunk publishes two excellent resources for learning and working with its search processing language (SPL):

When you start with developing apps for Splunk the first big question you hit is a bit unexpected: which technology do you work with, which app type do you choose?

If you have ever been to a conference with a Splunk booth you will have noticed their t-shirts. They are all black with a cool slogan printed in white. When you have been working with Splunk for some time you will notice that it has a very powerful commandline tool, splunk.exe. You can use that to tell Splunk’s deployment server component that one or more apps have changed, for example, with this command: splunk.exe reload deploy-server.

Splunk has a very extensive REST API - which is just a fancy way of saying that many of its capabilities are accessible via standard HTTP(S) requests. While much of the API is well documented, submitting data from C# to Splunk is kept a bit vague. Since I had to do that recently in order to implement an event generator for uberAgent I thought I might share my two solutions, both with and without the Splunk SDK for C#.

Processing several terabytes of data per day is not too uncommon and easily possible with Splunk - that is one of the many things I learned at .conf 2013 in Las Vegas.

Analyzing slow boots is a difficult task. You need to install software like XPerf and master its far-from-intuitive command-line options to generate a trace file that you can then analyze. Once you find a possible cause for the long startup duration you never know if it is specific to the machine you analyzed or if it affects other PCs, too. In other words: XPerf, although powerful, is difficult to master. And it does not scale. uberAgent does. And it is super-easy to use.

Do you know the maximum number of users each of your terminal servers can host with acceptable performance? You may have found out the hard way how many are too many - but how many are just right? Farm sizing and server capacity planning are typical tasks for consultants who often have a hard time fighting the peculiarities of perfmon and logman trying to get the data they need for their calculations. It can be so much easier at no additional cost. The 60-day Enterprise Trial version of Splunk in conjunction with an evaluation license of uberAgent give all the information you need in much less time. Here is how.

The days are long gone when a browser was just another application. Modern websites are applications of their own, and the browser is their operating system. That has consequences for monitoring. It is no longer sufficient to gather performance data for the browser as a whole. When, for example, Internet Explorer’s CPU usage is high, Administrators need to understand what caused that. Is it the business-critical ERP site or are people just watching fun videos on YouTube?