I’m using the kube-prometheus-stack Helm chart to deploy Prometheus and configure ServiceMonitors for monitoring EC2 instances with the Node Exporter. I have configured the ServiceMonitor to relabel EC2 instance tags, but the instance label is not being applied as expected.

I have verified the following:

• EC2 instances have the required tags.

• IAM roles and permissions are correctly set up.

• ServiceMonitor and Prometheus configurations include the required relabeling rules.

  Configuration:

  prometheus-node-exporter:
    resources:
      requests:
        memory: "64Mi"
        cpu: "100m"
    service:
      annotations:
        service.kubernetes.io/topology-aware-hints: Auto
        service.kubernetes.io/topology-aware-routing: Auto
    extraArgs:
      - --collector.ethtool
    nodeSelector:
      kubernetes.io/os: linux
    rbac:
      pspEnabled: false
    serviceMonitor:
      enabled: true
      interval: 30s
      jobLabel: node-exporter
      metricRelabelings: []
      relabelings:
        - sourceLabels: [__meta_ec2_tag_Name]
          action: replace
          targetLabel: instance
        - sourceLabels: [__address__]
          action: replace
          targetLabel: __address__
    prometheus:
      monitor:
        relabelings:
          - sourceLabels: [__meta_ec2_tag_Name]
            action: replace
            targetLabel: instance
          - sourceLabels: [__address__]
            action: replace
            targetLabel: __address__
  
  

  However, the relabeling for the instance label does not seem to be working. Below is the output of a dry run for the ServiceMonitor:

  apiVersion: monitoring.coreos.com/v1
  kind: ServiceMonitor
  metadata:
    name: monitoring-prometheus-node-exporter
    namespace: monitoring
    labels:
      helm.sh/chart: prometheus-node-exporter-4.34.0
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/component: metrics
      app.kubernetes.io/part-of: prometheus-node-exporter
      app.kubernetes.io/name: prometheus-node-exporter
      app.kubernetes.io/instance: monitoring
      app.kubernetes.io/version: "1.8.0"
      jobLabel: node-exporter
      release: monitoring
  spec:
    jobLabel: node-exporter
    selector:
      matchLabels:
        app.kubernetes.io/name: prometheus-node-exporter
        app.kubernetes.io/instance: monitoring
    attachMetadata:
      node: false
    endpoints:
      - port: http-metrics
        scheme: http
        relabelings:
          - action: replace
            sourceLabels:
            - __meta_ec2_tag_Name
            targetLabel: instance
          - action: replace
            sourceLabels:
            - __address__
            targetLabel: __address__
  
  

  Despite this configuration, the instance label does not appear in the Prometheus UI for the targets.

  I expected the instance label to be populated with the EC2 tag Name for each target monitored by the ServiceMonitor. However, this is not happening, and the label is not present in the target list in the Prometheus UI.