After upgrading Nginx ingress a few times we noticed that downloads with persistent session for files of 2+GB do not work with a 403 error saying that the file is not available. Timeline of after what change exactly it happened. Current version is version 4.10.1 of the helm chart. We come from version 4.4.0 on EKS 1.24 when it worked. Then to version 4.9.1 in version 1.27 and 1.28 of EKS and now 4.10.1 in version 1.28. We just noticed big files downloads were not working on version 4.9.1 and then we then upgraded to 4.10.1 to check if that was it aside from doing a lot of other test changes

Here is the log of the failure in Nginx:

10.1.14.79 - - [30/Apr/2024:03:30:58 +0000] "GET /URI/Archive%204.zip HTTP/1.1" 403 23 "https://sundomain.com/URI/My%20Library/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36" 734 0.001 [default-host-domain-com-80] [] 10.1.17.19:80 23 0.001 403 6cd138dda0809e2043dc96bf430dcfec

this is the nginx ingress controller config:

controller:
  config:
    enableSnippetAnnotations: true
    data:
      allow-snippet-annotations: 'true'
      http-snippet: |
        server{
          listen 2443;
          return 308 https://$host$request_uri;
        }
      use-forwarded-headers: 'true'
      map-hash-bucket-size: "256"
      server-name-hash-max-size: "1024"
      types-hash-max-size: "128"

From there is missing parts of the values file that are kind standard. We are on AWS and use NLB and they have just been working, so nothing has changed there.

Then the actual ingress looks like this:

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-subdomain-com-ingress
  labels:
    ingress-class: "ingress-class"
  annotations:
    # https://kubernetes.github.io/ingress-nginx/examples/affinity/cookie/
    kubernetes.io/ingress.class: "ingress-class"
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/affinity-mode: persistent
    nginx.ingress.kubernetes.io/session-cookie-name: "route"
    nginx.ingress.kubernetes.io/session-cookie-expires: "15000"
    nginx.ingress.kubernetes.io/session-cookie-max-age: "15000"
    nginx.ingress.kubernetes.io/session-cookie-change-on-failure: "true"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "10800"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "10800"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "10800"
    nginx.ingress.kubernetes.io/keep-alive: "250"
    nginx.ingress.kubernetes.io/client-body-timeout: "10800"
    nginx.ingress.kubernetes.io/client-max-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-buffering: "off"
    nginx.ingress.kubernetes.io/proxy-max-temp-file-size: "0"
    # nginx.ingress.kubernetes.io/ssl-redirect: "true"
    # nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
    # nginx.ingress.kubernetes.io/client-header-timeout: "120s"
    # nginx.ingress.kubernetes.io/proxy-body-size: "100000m"
    # nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
    # nginx.ingress.kubernetes.io/proxy-buffering: "off"
    # nginx.ingress.kubernetes.io/load-balance: "least_time"
spec:
  rules:
  - host: subdomain.com
    http:
      paths:
        - pathType: Prefix
          path: /
          backend:
            service:
              name: app-subdomain-com
              port:
                number: 80

everything else works as expected for instance I can upload a big file no issue, I just cant download on the browser I get the 403 with the message “file wasn’t available on site”. Notice the commented out parts are some of the thing I already tried and did not help too. Also with a regular nginx something like this works as expected:

upstream web {
    server myserver:80;
}

server {

    listen 80;

    location / {
        return 301 https://$host$request_uri;
    }

}

server {
    listen 443 ssl;
    ssl_certificate /etc/letsencrypt/live/site/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/site/privkey.pem;
  
    location / {
        proxy_pass http://web;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $remote_addr;
        
        client_max_body_size 0;
        proxy_connect_timeout  36000s;
        proxy_read_timeout  36000s;
        proxy_send_timeout  36000s;
        send_timeout  36000s;
    }
}

any ideas?