I am facing an issue with Kubernetes Horizontal Pod Autoscaler (HPA) which is unable to fetch custom metrics from the Prometheus Adapter. Despite following the configuration steps and verifying that metrics are available, the HPA reports no metrics.

Environment Details:

Kubernetes version: v1.28.2

Prometheus Adapter version: v0.11.2

Custom Metrics API is configured and available

Prometheus Adapter Configuration:

My values.yaml configuration for the Prometheus Adapter is as follows:

rules:
  default: false
  custom:
    - seriesQuery: 'app_horizon_current_workload{queue!=""}'
      resources:
        overrides:
          namespace:
            resource: "namespace"
          pod:
            resource: "pod"
      name:
        matches: "app_horizon_current_workload"
        as: "app_horizon_max_workload"
      metricsQuery: 'max(<<.Series>>{<<.LabelMatchers>>,queue!=""}) by (<<.GroupBy>>)'

Horizontal Pod Autoscaler (HPA) Configuration:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: hpa-queue-worker
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: queue-worker
  minReplicas: 4
  maxReplicas: 16
  metrics:
    - type: Pods
      pods:
        metric:
          name: app_horizon_max_workload
        target:
          type: AverageValue
          averageValue: "10"

RBAC Configuration:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: custom-metrics-reader
rules:
- apiGroups: ["custom.metrics.k8s.io"]
  resources: ["*"]
  verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: custom-metrics-reader-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: custom-metrics-reader
subjects:
- kind: ServiceAccount
  name: hpa-controller
  namespace: kube-system

Logs and Metrics Verification:

1. Prometheus Adapter Logs:

I0724 07:33:07.448719       1 httplog.go:132] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/app_horizon_max_workload" latency="36.854307ms" userAgent="kubectl/v1.30.2 (linux/amd64) kubernetes/3968350" audit-ID="9dbd3da1-3262-47f4-900e-37057a6545a6" srcIP="10.244.3.42:51390" resp=200

2. Prometheus Query:

Executing the Prometheus query max(app_horizon_current_workload{queue!=””}) by (namespace, pod) returns the expected results:

{
  "namespace": "default",
  "pod": "pod-1"
}
22
{
  "namespace": "default",
  "pod": "pod-2"
}
24

3. kubectl Custom Metrics API:

Running kubectl get --raw '/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/app_horizon_max_workload' | jq . I get:

{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {},
  "items": [
    {
      "describedObject": {
        "kind": "Pod",
        "namespace": "default",
        "name": "pod-1",
        "apiVersion": "/v1"
      },
      "metricName": "app_horizon_max_workload",
      "timestamp": "2024-07-24T07:33:07Z",
      "value": "22",
      "selector": null
    },
    {
      "describedObject": {
        "kind": "Pod",
        "namespace": "default",
        "name": "pod-2",
        "apiVersion": "/v1"
      },
      "metricName": "app_horizon_max_workload",
      "timestamp": "2024-07-24T07:33:07Z",
      "value": "24",
      "selector": null
    }
  ]
}

Issue:

Despite the custom metrics being available, the HPA reports <unknown> for the metrics and fails to scale:

Running kubectl describe hpa hpa-queue-worker -n default I get:

Name:                                  hpa-queue-worker
Namespace:                             default
Metrics:                               ( current / target )
  "app_horizon_max_workload" on pods:  <unknown> / 10
Conditions:
  Type           Status  Reason               Message
  ----           ------  ------               -------
  AbleToScale    True    SucceededGetScale    the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetPodsMetric  the HPA was unable to compute the replica count: unable to get metric app_horizon_max_workload: no metrics returned from custom metrics API
Events:
  Warning  FailedGetPodsMetric           0s (x2 over 15s)  horizontal-pod-autoscaler  unable to get metric app_horizon_max_workload: no metrics returned from custom metrics API
  Warning  FailedComputeMetricsReplicas  0s (x2 over 15s)  horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get pods metric value: unable to get metric app_horizon_max_workload: no metrics returned from custom metrics API

Question:
What might be causing the HPA to fail to fetch custom metrics from the Prometheus Adapter? How can I resolve this issue to enable the HPA to scale based on custom metrics?

Any help or suggestions would be greatly appreciated. Thank you!