Calculating Unique System Counts per Error Message with Loki Query in Grafana Dashboard

Question

I am currently working on a Grafana dashboard where I visualize various errors logged in different systems and count the number of times these errors occur. I am using Loki to send journal logs to Grafana. I've managed to create a query that groups error messages and counts their occurrences over a span of 7 days.

Here is my current Loki query:

topk(10, sum by(message)(count_over_time({job="systemd-journal"} |~ `ERROR:` | regexp `(?P<message>ERROR:.*)` [7d])))

The output from this query is as follows:

Now, I would like to extend this functionality to include an additional value which indicates from how many different systems these errors are coming. Ideally, this would be displayed next to the current value in the dashboard.

I tried adding a "Group by" transformation in Grafana, grouping by the message field and counting unique system identifiers (like hostname) associated with each error message. Here was my attempted query modification:

sum by(message, hostname)(count_over_time({job="systemd-journal"} |~ `ERROR:` | regexp `(?P<message>ERROR:.*)` [7d]))

I expected to see an additional column indicating the count of unique systems per error message. However, this approach doesn't seem to work, as I end up with no data when adding the "Group by" transformation in Grafana.

My output for the query above without the group by transformation:

So without group by transformation your query works as expected, but transformation produces unexpected results? — markalex
– markalex, Commented Sep 18, 2023 at 19:40
I have posted a screenshot of the output without the 'group by' transformation. As you can see, the hostname is displayed on the left side, and the specific count of the unique errors for the host is on the right side. What I actually want, though, is just the count of the systems that have this unique error. So for example, if 3 systems have the error "ERROR: No reversePort received [0m", I would like to have an output like this: "ERROR: No reversePort received[0m 21 3" When I add the 'group by' message transformation, I get "no data" as an output. — Falke
– Falke, Commented Sep 19, 2023 at 6:02
Aha, so you don't want a list of hosts, just number of them! Then you can simply use count by(hostname) ( <your attempt sum by(message, hostname) ...> ) without any transformation. Then you can apply Join by field to join results of first and second queries. (Don't forget to set both of them to type Instant) — markalex
– markalex, Commented Sep 19, 2023 at 8:06

Falke · Accepted Answer · 2023-09-23 16:23:52Z

0

Solved it with markalex suggestion count by(hostname) ( <your attempt sum by(message, hostname) ...> )

answered Sep 23, 2023 at 16:23

Falke

1831 gold badge3 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Calculating Unique System Counts per Error Message with Loki Query in Grafana Dashboard

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related