Dashboard variable in grafana: ignore changing labels

Question

I have a dashboard in Grafana which shows Prometheus metrics for instances, where "instance" is operation system on physical or virtual server.

Number of instances is changing over time, so there is no "closed list of instances".

Dashboard detects list of instances on the fly with dashboard variable, "type" is "query", definition is label_values(node_uname_info{job="$job"}, instance). It works fine while set of labels for each "instance" in metric "node_uname_info" is unchanged. But if I add new label to metric, this results in "disappearance" of the old instance and "emergence" of a new instance.

Step 1: add instance "myhostname" as new prometheus target
Step 2: it appears in dashboard as new instance {instance="myhostname", job="myjob"}
Step 3: add label "newlabel" to existing prometheus target
Step 4: it appears in dashboard as new instance {instance="myhostname", job="myjob", newlabel="value"} (instead of old one)
Step 5: visualizations on dashboard shows metrics for "old" instance from step 2 and "new" instance from step 4 separately (with separate legends, different colors, etc.), because for Grafana these two instances are not the same

How could I change Grafana dashboard settings (I guess - settings related to dashboard variable) to completely ignore all labels, except job and instance, while detecting list of instances on the fly? So changing (or adding or removing) labels, except job and instance, will not cause in changing list of instances?

Jan Garaj · Accepted Answer · 2023-10-31 08:37:46Z

1

Each timeserie is defined by set of labels. Your panel query returns 2 timeseries in this case, so Grafana shows them as 2 different instances.

Your panel query should handle it (it is not a problem. of dashboard variable) - aggregate there per job, instance as well, so only single timeserie will be a result.

answered Oct 31, 2023 at 8:37

Jan Garaj

29k3 gold badges49 silver badges73 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

AntonioK Over a year ago

"aggregate there per job, instance as well" - Is replacing node_filesystem_avail_bytes{instance="$node",job="$job",device!~'rootfs'} with

sum by (job,instance,device,mountpoint,fstype) (node_filesystem_avail_bytes{instance="$node",job="$job",device!~'rootfs'})

the correct implementation of your answer?

Jan Garaj Over a year ago

@AntonioK I don't know your datamodel, so I can just guess: yes

Collectives™ on Stack Overflow

Dashboard variable in grafana: ignore changing labels

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related