so I am having issues whitelisting a IP range to allow access to the wp-admin endpoint under location.

I have something similar to the below in the nginx.conf (The IP listed is an example). When I just have deny all in /wp-admin I get a 403 as expected. When adding the allow subnet however I am met with a 404.

I also tried allowing a single IP but I am getting the same error.

location / {
      proxy_pass http://wordpress;
      
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $remote_addr;
      proxy_set_header Host $host;
    }
location /wp-admin {
      allow 123.0.0.0/4;
      deny all; 
    }

What is going wrong in the config that the allow function is not working correctly?

Thanks