Im trying to configure mosquitto MQTT failover configuration in nginx ingress. Im using a tcp port 8883 and then I want to provide custom upstream like below

upstream mqtt_cluster {
   server osc-mqtt-admin.osc.svc.cluster.local:8883 max_fails=1 fail_timeout=10s;
   server osc-mqtt-backup.osc.svc.cluster.local:8883 backup;
}

Exposing TCP services through nginx ingress creates a block like below in nginx.conf

server {
  preread_by_lua_block {
    ngx.var.proxy_upstream_name="tcp-osc-osc-mqtt-admin-8883";
  }
        
  listen                        8883;
  listen                        [::]:8883;
        
  proxy_timeout                 600s;
  proxy_next_upstream           on;
  proxy_next_upstream_timeout   600s;
  proxy_next_upstream_tries     3;
        
  proxy_pass                    upstream_balancer;      
}

In the above conf block proxy_pass is using upstream_balancer, instead I want it to use mqtt_cluster custom upstream

If anyone has this configured before please do guide me.

Thanks

I tried using ingress annotation

nginx.ingress.kubernetes.io/stream-snippet: |
   log_format mqtt '$remote_addr [$time_local] $protocol $status $bytes_received '
      '$bytes_sent $upstream_addr';

      error_log stdout;

      upstream mqtt_cluster {
          server osc-mqtt-admin.osc.svc.cluster.local:8883 max_fails=1 fail_timeout=10s;
          server osc-mqtt-backup.osc.svc.cluster.local:8883 backup;
      }

      server {
          listen 8883;
          proxy_connect_timeout 1s;
          proxy_pass mqtt_cluster;
          proxy_buffer_size 16k;
      }

This is applied but ingress controller service is not exposed with 8883 port. If I configure tcp service along with the I get duplicate 0.0.0.0:8883 configuration error.