I am creating a Grafana dashboard to monitor gRPC API servers A, B, and C. All these servers are deployed in their individual containers on a k8s cluster and each have CRUD endpoints. The names of the endpoints follow the naming convention below:

my.grpc.service.servers.v1.{server-name}Create
my.grpc.service.servers.v1.{server-name}Read
my.grpc.service.servers.v1.{server-name}Update
my.grpc.service.servers.v1.{server-name}Delete

where {server-name} is the name of the servers i.e. A, B, or C.

I have a Grafana dashboard with three variables:

1. Cluster: This is a custom variable representing names of the k8s cluster & is a hardcoded list.
2. Namespace: This is a custom variable representing names of the k8s namespace & and is a hardcoded list.
3. Container: This is a query variable label_values(kube_pod_container_info{namespace=~$namespace}, container) and is a drop down list on the dashboard with ability to select multi values. The container names for server A, B, and C are container-A, container-B, and container-C, respectively.

This dashboard has a row with 2 panels that replicate themselves for the container. So, if I select more than one container in the Container drop down list, there will be twice the number of panels in the row (one container selected = 2 panels, 2 containers selected = 2 * 2 panels, and so on). One of the panel is to show the cpu usage (container_cpu_usage_seconds_total) and another is to show the memory usage (container_memory_working_set_bytes + container_memory_rss).

Now, I want to add one more panel to this row for a custom metric, let’s call it api_usage_total, which has a label named endpoint_name. I want the value for the label endpoint_name to be set based on the containers selected in drop down list. For example:

• If I select container-A, the value of this label should be my.grpc.service.servers.v1.A.+.
• If I select container-A and container-C, there should be two such panels each having endpoint_name set as my.grpc.service.servers.v1.A.+ and my.grpc.service.servers.v1.C.+, respectively.

Please note that there are no common labels between the k8s metrics and the custom metric.

I have tried:

• to look for an if-else logic in PromQL but could not find any helpful documents.
• to set key:value pairs in a new variable but it did not work since this new variable is not linked to the container variable in any way.

Please help.

PS: Please note that I can’t share the screenshots due to corporate restrictions.