1

I am creating a Grafana dashboard to monitor gRPC API servers A, B, and C. All these servers are deployed in their individual containers on a k8s cluster and each have CRUD endpoints. The names of the endpoints follow the naming convention below:

my.grpc.service.servers.v1.{server-name}\Create
my.grpc.service.servers.v1.{server-name}\Read
my.grpc.service.servers.v1.{server-name}\Update
my.grpc.service.servers.v1.{server-name}\Delete

where {server-name} is the name of the servers i.e. A, B, or C.

I have a Grafana dashboard with three variables:

  1. Cluster: This is a custom variable representing names of the k8s cluster & is a hardcoded list.
  2. Namespace: This is a custom variable representing names of the k8s namespace & and is a hardcoded list.
  3. Container: This is a query variable label_values(kube_pod_container_info{namespace=~$namespace}, container) and is a drop down list on the dashboard with ability to select multi values. The container names for server A, B, and C are container-A, container-B, and container-C, respectively.

This dashboard has a row with 2 panels that replicate themselves for the container. So, if I select more than one container in the Container drop down list, there will be twice the number of panels in the row (one container selected = 2 panels, 2 containers selected = 2 * 2 panels, and so on). One of the panel is to show the cpu usage (container_cpu_usage_seconds_total) and another is to show the memory usage (container_memory_working_set_bytes + container_memory_rss).

Now, I want to add one more panel to this row for a custom metric, let's call it api_usage_total, which has a label named endpoint_name. I want the value for the label endpoint_name to be set based on the containers selected in drop down list. For example:

  • If I select container-A, the value of this label should be my.grpc.service.servers.v1.A\.+.
  • If I select container-A and container-C, there should be two such panels each having endpoint_name set as my.grpc.service.servers.v1.A\.+ and my.grpc.service.servers.v1.C\.+, respectively.

Please note that there are no common labels between the k8s metrics and the custom metric.

I have tried:

  • to look for an if-else logic in PromQL but could not find any helpful documents.
  • to set key:value pairs in a new variable but it did not work since this new variable is not linked to the container variable in any way.

Please help.

PS: Please note that I can't share the screenshots due to corporate restrictions.

Edit: There was a mistake in the example regarding container names. In reality, they don't follow any naming convention. Let's say the container names are A-container, B-server, and C-service-server.

Improve this question

1 Answer 1

Reset to default
0

The fact that you are using multi-value variable and row repetition makes things more complicated.

For your case I recommend following path:

  1. Introduce additional label into metric, that would contain name of the service in format it is present in your variable list:
label_replace(api_usage_total, "container_name", "container-$1", "endpoint_name", "my\\.grpc\\.service\\.servers\\.v1\\.([^.]+).*")
  1. Filter out result based on the content of variable
label_replace(api_usage_total, "container_name", "container-$1", "endpoint_name", "my\\.grpc\\.service\\.servers\\.v1\\.([^.]+).*")
 and on(container_name) absent(_{container_name="$container")

What happens here: since label container_name is synthetic, and was introduced by label_replace we cannot filter by it directly. So we utilize a little trick: absent(_{container_name="$container") returns a time series with label container_name set to value of variable $container* (more about absent trying to be clever in documentation), and then using logical binary operator we apply filter to the left part to leave only time series having same label container_name.

* : In case of row or panel repetition variable is automatically replaced with a single item assigned to specific row. So in your example even though two values were selected, inside query of each panel $container with be replaced with container-A and container-C in each row respectively.

Some additional information: if you were not to use row repetition and multi-value variable (in a different dashboard with different expectations), same problem might be solved way easier with introduction of a "chained" variable. Basically you can create a separate variable, that would contain A if you selected container-A in variable container_name, and then use that new variable in a more straight forward fashion in queries.

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

Thank you @markalex for the response. I shall try it. However, I think I made a mistake in my examples with container names (made the edit to the original question), sorry about that. Would the solution still work?
@Anonymous, well if you know all possible pairs A -> container name, you still xan do that. It's just that you'll basically need to use not 1 label_replace but n+1, where n number of templates for container. If it's bigger that a couple or dynamic - described solution will be hard to maintain.
Alternatively, you could store pairs in artificial metric in prometheus (for example using record rules or text file exporter). And use and on two times.
No, its impossible to know container names in advance because new services get onboarded and development team has autonomy on choosing the container names and I'd not prefer imposing a naming convention on them. Basically, the idea behind this dashboard is make it once and done. New services will automatically get their panels if their containers are selected from the input drop down. I'd not mind having a mapping somewhere that gets updated (or that needs to be updated manually) whenever new services are deployed. Would you please be able to help with that?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.