LogQL: A Primer on Querying Loki from Grafana

This article introduces newbies to writing search queries in LogQL. It should help you get started with building Grafana dashboards based on log data in Loki. It would have helped me immensely building my first Grafana dashboard.

Learning Grafana & Loki as a Splunk User

If you’re coming from a Splunk background, like I do, you’re in for a surprise, and not an altogether pleasant one. As expected, things are different. But not only that: they’re often more convoluted than they need to be. It doesn’t help that Grafana’s documentation could be better at introducing newbies to the product and its quirks.

LogQL Primer: Building a Query

LogQL, Loki’s query language, is a mish-mash of different concepts. This makes queries harder to understand and, of course, to write, than necessary.

Log Stream Selector

In principle, though, LogQL is simple. A query starts with a log stream selector that specifies which labels to search, e.g., the label resticprofile:

{job="resticprofile"}

Line Filter

The log stream selector is followed by optional line filter expressions, which act like a grep on the log line to narrow the search to those events you’re interested in, e.g., lines containing the string Files::

|= `Files:`

Parser: Label Extraction

A parser extracts labels (another word for fields) from the log line. JSON and key-value (logfmt) formats can be parsed automatically. If your logs are formatted differently but have a consistent structure, the pattern parser might fit. Otherwise, you need to resort to regular expressions. The following regexp parser example extracts a number into the label new:

# Log line sample:
# Files:          63 new,    60 changed, 695354 unmodified
| regexp `Files:\\s+(?P<new>\\d+)\\s+new,`

Label Format Expression: Calculations

Label format expressions are roughly equivalent in nature to Splunk’s all-purpose eval command. It took me a while to discover them; make sure to read their documentation and look at the template functions that can be used with them.

The following label_format example adds the values of the labels new and changed and assigns the result to the new label modified:

| label_format modified=`{{add .new .changed}}`

Unwrap: Logs to Metrics

To perform a metrics calculation like sum or average on a label, the label needs to be unwrapped. Incomprehensibly, this can only be done for a single label per query. If you need multiple data series, you need to write and run one query per label. Not exactly efficient.

The following unwrap example extracts a metric from the label modified:

| unwrap modified [$__auto]

Unwrapped Range Aggregation: Metrics Query on Labels

Statistical aggregations are the bread and butter of a visualization product. They massage the data into a format suitable for display as a (time) chart. In Splunk, the stats and timechart aggregation commands are appended to the pipeline. In Grafana, the *_over_time aggregation command is wrapped around the entire query, which hurts readability a lot.

The following full example calculates the sum_over_time for the label ratio:

sum_over_time(
  {$job_label_name="$job_label_value", host=~"(${host:pipe})"}
    |= `Files:`
    | regexp `Files:\s+(?P<new>\d+)\s+new,\s+(?P<changed>\d+)\s+changed,\s+(?P<unmodified>\d+)\s+unmodified`
    | label_format modified=`{{add .new .changed}}`
    | label_format ratio=`{{divf .modified .unmodified}}`
    | drop new, changed, modified, unmodified
    | unwrap ratio [$__auto]
)

LogQL Tips

Remove Labels You Don’t Need

As there is no way to select which labels to use for charting in Grafana, the LogQL query must not contain unnecessary labels. Only give Grafana what it should draw.

Removing labels can be done with the drop command as shown in the example above. I would generally prefer to specify which labels to keep but couldn’t get that command to work.

Grafana Tips

Rename the Charted Field/Series

LogQL queries generate series names based on the labels’ values, which is not helpful for charting. Make use of Grafana’s renaming capability to set a suitable display name. To do so, navigate to Edit panel > Standard options > Display name.

An example from my restic backup monitoring dashboard:

• Original series name: {filename="/promtail/collect/resticprofile/backup.log", host="px1", job="resticprofile"}
• Renamed series name: size