I have the following Prom scraper config for scraping cadvisor metrics (plus pod_exporter)

global:
  scrape_interval: 30s
  external_labels:
    cluster: foo-bar

scrape_configs:
  - job_name: pod_exporter
    kubernetes_sd_configs:
      - role: pod

  - job_name: cadvisor
    scheme: https
    authorization:
      type: Bearer
      credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    kubernetes_sd_configs:
      - role: node
    relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - action: labeldrop
        regex: (feature_node_kubernetes_io.*|nvidia_com.*|failure_domain_beta_kubernetes_io.*)
      - replacement: kubernetes.default.svc:443
        target_label: __address__
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor

I’m trying to drop some node labels from the scraped metrics to avoid storing unnecessary data – we don’t need those metrics dimensions anyway.

I was hoping the labeldrop action as listed in the config would drop the node labels that look like the following:

feature.node.kubernetes.io/....
failure-domain.beta.kubernetes.io/...
nvidia.com/...

I’m sure I don’t quite understand in detail how the labelmap action works, though, which leads to those node labels not being dropped by the scraper.

My understanding is it does the following:

1. __meta_kubernetes_node_label_feature_node_kubernetes_io_... is added automagically and then in the next step, before the labeldrop kicks in the __meta* part is stripped and the labeldrop actually see these labels:
2. feature_node_kubernetes_io...

So I figured this regex in the labeldrop action should drop them

      - action: labeldrop
        regex: (feature_node_kubernetes_io.*|nvidia_com.*|failure_domain_beta_kubernetes_io.*)

But that doesn’t seem to be the case. Does anyone know how to drop the labels I mentioned above?