I have Traefik deployed with following config

  traefik.toml: |
    ## static configuration
    [global]
      checkNewVersion = true

    [entryPoints]
      [entryPoints.web]
        address = ":80"
      [entryPoints.websecure]
        address = ":443"

    [providers]
      [providers.kubernetesCRD]
      [providers.file]
        directory = "/etc/traefik/providers/"
        watch = true
      [providers.kubernetesIngress]
        ingressClass = "traefik-cert-manager"

    [log]
      level = "INFO"

    [accessLog]

    [api]
      insecure = true
      dashboard = true
      debug = true

    [metrics]
      [metrics.prometheus]
        buckets = [0.1,0.3,1.2,5.0]
        addEntryPointsLabels = true
        addServicesLabels = true

and Deployment Manifest

kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: traefik
  namespace: kube-system
  labels:
    app: traefik
  annotations:
     prometheus.io/scrape: "true"
     prometheus.io/port: "8080"
spec:
  selector:
    matchLabels:
      app: traefik
  minReadySeconds: 5
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  template:
    metadata:
      labels:
        app: traefik
    spec:
      serviceAccountName: traefik-ingress-controller
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      containers:
      - name: traefik
        image: traefik:2.9.6
        ports:
          - name: web
            containerPort: 80
          - name: websecure
            containerPort: 443
          - name: internal
            containerPort: 4080
          - name: internalsecure
            containerPort: 4043
          - name: admin
            containerPort: 8080
        readinessProbe:
          failureThreshold: 1
          httpGet:
            path: /ping
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        livenessProbe: 
          httpGet: 
            path: /ping 
            port: 8080 
          failureThreshold: 5 
          initialDelaySeconds: 10 
          periodSeconds: 10 
          successThreshold: 1 
          timeoutSeconds: 2 
        securityContext:
          privileged: true
        env:
          - name: AWS_REGION
            valueFrom:
              configMapKeyRef:
                name: aws-config
                key: aws_region
          - name: AWS_HOSTED_ZONE_ID
            valueFrom:
              configMapKeyRef:
                name: aws-config
                key: aws_hosted_zone_id
          - name: AWS_ACCESS_KEY_ID
            valueFrom:
              secretKeyRef:
                name: aws-secret
                key: access_key
          - name: AWS_SECRET_ACCESS_KEY
            valueFrom:
              secretKeyRef:
                name: aws-secret
                key: secret_key
        volumeMounts:
          - name: traefik-config
            mountPath: /etc/traefik/traefik.toml
            subPath: traefik.toml
          - name: traefik-config
            mountPath: /etc/traefik/providers/dynamic.toml
            subPath: dynamic.toml
          - name: traefik-storage
            mountPath: /etc/traefik/storage/
      volumes:
      - name: traefik-config
        configMap:
          name: traefik-conf
      - name: traefik-storage
        hostPath:
          path: /tmp/traefik
          type: DirectoryOrCreate

With this I can see metrics at /metrics for Traefik.

But unable to get it into Prometheus with following scrap job

      - job_name: 'traefik'
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_label_app]
            action: keep
            regex: traefik
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
            action: keep
            regex: true
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels: [__address__]
            action: replace
            target_label: __address__
            regex: ([^:]+)(?::d+)?;(d+)
            replacement: $1:9100

What I am missing here ?