We have been trying to setup a prometheus on k8s using the prometheus community helm chart. From what I understand to enable basic auth for the prometheus dashboard it is only really possible to setup the basic auth on the ingress. We struggled to setup the ingress independently of this helm, only when we enable ingress on the values.yaml and fed that to helm, did the ingress work (so the chart did the setup). So we can access the prometheus dashboard no issue. But when following the config for the basic auth using the nginx ingress nothing happens, we added the annotations to the prometheus values.yaml and upgraded the install. Viewing the ingress after the update shows the annotations where added but basic auth is still disabled. If we view the nginx ingress controller pod logs we see the request come in but no attempt to prompt for auth, its not even trying.

Secret was created as per the nginx ingress documentation, but even if this was wrong or missing I would expect that basic auth would still attempt a challenge, but nothing.

This this what we added in the prometheus values.yaml for ingress

   ingressClassName: nginx

    ## Prometheus server Ingress annotations
    ##
    annotations: 
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/auth-realm: Authentication Required
      nginx.ingress.kubernetes.io/auth-secret: monitoring/basic-auth
      nginx.ingress.kubernetes.io/auth-type: basic
      cert-manager.io/issuer: "selfsigned-issuer"
        
    
    #   kubernetes.io/ingress.class: nginx
    #   kubernetes.io/tls-acme: 'true'

    ## Prometheus server Ingress additional labels
    ##
    extraLabels: {}

    ## Redirect ingress to an additional defined port on the service
    # servicePort: 8081

    ## Prometheus server Ingress hostnames with optional path
    ## Must be provided if Ingress is enabled
    ##
    hosts:
        - prometheus.sandbox.io
    #   - prometheus.domain.com
    #   - domain.com/prometheus

    path: /

    # pathType is only for k8s >= 1.18
    pathType: Prefix

    ## Extra paths to prepend to every host configuration. This is useful when working with annotation based services.
    extraPaths: []
    # - path: /*
    #   backend:
    #     serviceName: ssl-redirect
    #     servicePort: use-annotation

    ## Prometheus server Ingress TLS configuration
    ## Secrets must be manually created in the namespace
    ##
    tls:
      - secretName: prometheus-server-tls
        hosts:
          - prometheus.sandbox.io

We have cert manager installed and using a self-signed issuer for testing, this works fine with the usual cert warnings.

This is the ingress itself:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/issuer: selfsigned-issuer
    kubernetes.io/ingress.class: nginx
    meta.helm.sh/release-name: prometheus
    meta.helm.sh/release-namespace: monitoring
    nginx.ingress.kubernetes.io/auth-realm: Authentication Required
    nginx.ingress.kubernetes.io/auth-secret: monitoring/basic-auth
    nginx.ingress.kubernetes.io/auth-type: basic
  creationTimestamp: "2024-08-02T08:23:02Z"
  generation: 1
  labels:
    app.kubernetes.io/component: server
    app.kubernetes.io/instance: prometheus
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/part-of: prometheus
    app.kubernetes.io/version: v2.53.1
    helm.sh/chart: prometheus-25.24.1
  name: prometheus-server
  namespace: monitoring
  resourceVersion: "1198730"
  uid: b1382534-4e4a-4eb3-a584-8f50ab70ceda
spec:
  ingressClassName: nginx
  rules:
  - host: prometheus.sandbox.io
    http:
      paths:
      - backend:
          service:
            name: prometheus-server
            port:
              number: 80
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - prometheus.sandbox.io
    secretName: prometheus-server-tls
status:
  loadBalancer:
    ingress:
    - ip: <public ip to access>

So kind stumped at the moment, if anyone has a method to do this or can advise of where we going wrong it would be appreciated.

Have tried to recreate the ingress, recreate the secret, added the namespace for the secret in the annotation. I assume that the ingress is not recognising the annotations at all as even if the secret was wrong the challenge would still happen but nothing seems to happen