We are running into an issue with the HAProxy ingress controller and NVIDIA Triton when using GRPC to communicate between client and server. When using HTTP the requests are passed through from the client to the Triton instance without issue. When using GRPC we get an HTTP 302 error. HAPproxy is configured as follows:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ template "triton-inference-server.fullname" . }}
  namespace: {{ .Release.Namespace }}
  labels:
    app: {{ template "triton-inference-server.name" . }}
    chart: {{ template "triton-inference-server.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
  annotations:
    haproxy-ingress.github.io/oauth: "oauth2-proxy"
    haproxy-ingress.github.io/oauth-uri-prefix: "/oauth2"
    haproxy-ingress.github.io/cors-enable: "true"
    haproxy-ingress.github.io/proxy-body-size: "256m"
    haproxy-ingress.github.io/timeout-client: "480"
    haproxy-ingress.github.io/timeout-server: "480"    
spec:
  ingressClassName: haproxy
  tls:
    - hosts:
        - {{ .Values.hostName }}
      secretName: {{ .Release.Name }}-tls
  rules:
    - host: {{ .Values.hostName }}
      http:
        paths:
        - backend:
            service:
              name: {{ template "triton-inference-server.fullname" . }}
              port:
                number: 8000
          path: /
          pathType: ImplementationSpecific
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    # haproxy-ingress.github.io/redirect-to: "https://{{ .Values.hostName }}/inference.GRPCInferenceService"
    haproxy-ingress.github.io/oauth: "oauth2-proxy"
    haproxy-ingress.github.io/oauth-uri-prefix: "/oauth2"
    haproxy-ingress.github.io/cors-enable: "true"
    haproxy-ingress.github.io/backend-protocol: "h2"
    haproxy-ingress.github.io/proxy-body-size: "256m"
    haproxy-ingress.github.io/timeout-client: "480"
    haproxy-ingress.github.io/timeout-server: "480"
    haproxy-ingress.github.io/app-root: "/inference.GRPCInferenceService"
  name: {{ template "triton-inference-server.fullname" . }}-ingress-grpc
  namespace: {{ .Release.Namespace }}
spec:
  ingressClassName: haproxy
  tls:
  - hosts:
      - {{ .Values.hostName }}
    secretName: {{ .Release.Name }}-tls
  rules:
  - host: {{ .Values.hostName }}
    http:
      paths:
      - backend:
          service:
            name: {{ template "triton-inference-server.fullname" . }}
            port:
              number: 8001
        path: /inference.GRPCInferenceService
        pathType: Prefix

I have seen others struggle with this in

How to configure haproxy-ingress for serving GRPC

and

HAProxy Ingress Controller regarding mixed support of HTTP2 and HTTP 1.1 in the backend with TLS in frontend

but most of these are pretty old and don’t quite address our issue.

How are we misconfigurating the haproxy-ingress?

Thanks!