32

I'm running prometheus inside kubernetes cluster.

I need to send queries to Prometheus every minute, to gather information of many metrics from many containers. There are too match queries, so I must combine them.

I know how I can ask Prometheus for one metric information on multiple containers: my_metric{container_name=~"frontend|backend|db"} , but I haven't found a way to ask Prometheus for multiple metric information in one query.

I'm looking for the equivalent to 'union' in sql queries.

Improve this question

3 Answers 3

Reset to default
52

I found here this solution: {__name__=~"metricA|metricB|metricC",container_name=~"frontend|backend|db"}.

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

The article you link to specifically says that using regex on __name__ is an antipattern.
prometheus.io/docs/prometheus/latest/querying/basics states no qualms against this construct.
The linked article says that its example regex with wildcards "is an anti-pattern, as you're meant to know your metric names a-priori." I think the spirit of this question and answer is that you do know the exact names of the metrics you want (that is, you're not using any wild cards in your regex) you just want more than one at a time. To me, this answer doesn't seem to fit into the anti-pattern they're worried about.
I came here looking how to supply a list of actual (complex) queries, not just simple metrics.
7

You can use the or operator, however this does not generalise as it ignores metric names. I'd suggest making multiple queries to the API.

Improve this answer

1 Comment

Thanks, easiest and most elegant solution for quickly debugging a query in Prometheus' query tool.
5

Prometheus provides or operator, which can be used for combining results from multiple queries in a single API call. For example, the following query returns results for q1, q2 and q3:

q1 or q2 or q3

The or operator in PromQL has a quirk though - if some time series returned by q1, q2 and q3 have the same set of labels (excluding metric name aka __name__ label), then only the first encountered time series is returned. For example, the following query returns only process_cpu_seconds_total time series, and doesn't return process_resident_memory_bytes time series, since they have the same set of labels except of metric name:

process_cpu_seconds_total or process_resident_memory_bytes

This can be fixed by copying the metric name into another label. For example, the following query copies the metric name into metric_name label, so the or result contains time series from both process_cpu_seconds_total and process_resident_memory_bytes sides:

label_join(process_cpu_seconds_total, "metric_name", "", "__name__")
  or
label_join(process_resident_memory_bytes, "metric_name", "", "__name__")

P.S. There is a simpler solution for obtaining results from multiple queries in one call if using an alternative Prometheus-like monitoring system I work on - VictoriaMetrics - it provides union function for this task. For example, the following query returns time series for both process_cpu_seconds_total and process_resident_memory_bytes queries:

union(
  process_cpu_seconds_total,
  process_resident_memory_bytes,
)
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.