I have a private GKE cluster. Egress is configured with Cloud NAT. Ingress through Load Balancers.
Egress traffic costs me a lot, so I’d like to add a VM inside the network and configure IP forwarding and NAT there. After that, I will add custom routes to VPC with a few destinations that I use the most. The traffic should go through the NAT instance, and with lower costs.

So far, I have a VM with Ubuntu 22.04 created inside GKE’s network and with public IP attached (and also can-ip-forward enabled). Executed commands:

sysctl -w net.ipv4.ip_forward=1
iptables -t nat -A POSTROUTING -o ens4 -j MASQUERADE 

Added route to VPC (93.184.215.14 is the IP of website example.com – for testing):

gcloud compute routes create test-route --network my-gke-net 
  --destination-range 93.184.215.14/32 
  --next-hop-instance my-nat-instance 
  --next-hop-instance-zone us-central1-a 
  --priority 900

But when I’m trying to curl example.com it stucks.
On NAT instance I see some traffic in tcpdump -i ens4 'port 80' and journalctl -f shows IPv4: martian source 93.184.215.14 from 10.0.1.12, on dev ens4.
Firewall ingress and egress for NAT instance are allowed on all TCP ports.

Do you have any idea why this is not working?