I am using the Prometheus Helm chart in my Kubernetes cluster to monitor services. However, I am unable to get Prometheus to discover or scrape the LoadBalancer Ingress field of a Kubernetes service.

Here is the configuration of the service:

Name:                     postgres-xyz-postgresql-primary
Namespace:                xyz
Labels:                   app.kubernetes.io/component=primary
                      app.kubernetes.io/instance=postgres-app
                      app.kubernetes.io/managed-by=Helm
                      app.kubernetes.io/name=postgresql
                      app.kubernetes.io/version=17.0.0
                      helm.sh/chart=postgresql-16.0.1
Annotations:              meta.helm.sh/release-name: postgres-app
                      meta.helm.sh/release-namespace: xyz
Selector:                               app.kubernetes.io/component=primary,app.kubernetes.io/instance=postgres-   app,app.kubernetes.io/name=postgresql
Type:                     LoadBalancer
IP Family Policy:         SingleStack  
IP Families:              IPv4
IP:                       
IPs:                      
LoadBalancer Ingress:     aoo000nnbbgghhsdfg-182496858.us-east-1.elb.amazonaws.com
Port:                     tcp-postgresql  5432/TCP
 TargetPort:               tcp-postgresql/TCP
 NodePort:                 tcp-postgresql  32599/TCP
Endpoints:                10.0.91.92:5432
Session Affinity:         None
External Traffic Policy:  Cluster

In my Prometheus scrape configuration, I am using the following job for service discovery:

- job_name: 'kubernetes-services'
  honor_labels: true

  metrics_path: /probe
  params:
    module: [http_2xx]

 kubernetes_sd_configs:
   - role: service

 relabel_configs:
   - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
     action: keep
     regex: true
   - source_labels: [__address__]
     target_label: __param_target
   - target_label: __address__
     replacement: blackbox
   - source_labels: [__param_target]
     target_label: instance
   - action: labelmap
     regex: __meta_kubernetes_service_label_(.+)
   - source_labels: [__meta_kubernetes_namespace]
     target_label: namespace
   - source_labels: [__meta_kubernetes_service_name]
     target_label: service

The scrape job does not seem to capture the LoadBalancer Ingress field.

I tried adding the following to the relabel_configs to map the LoadBalancer Ingress to a custom label (loadbalancer_ingress):

- source_labels: [__meta_kubernetes_service_annotation_external_dns_hostname]
  target_label: loadbalancer_ingress

However, this doesn’t seem to capture the LoadBalancer Ingress value

Questions:

1. How can I modify the Prometheus relabel_configs to include the LoadBalancer Ingress field for scraping?

2. Is there a specific label or field in the kubernetes_sd_configs that maps to the LoadBalancer Ingress field?

3. Do I need a separate configuration to target LoadBalancer services explicitly?

Additional Info:

Kubernetes version: 1.30
Prometheus Helm chart version: prometheus-26.0.0

Any help in resolving this issue would be greatly appreciated!