We have a DNS problem in our on-oprem cluster, hopefully you can help.

In our node (ubuntu os) when running sudo resolvectl status the current DNS Server kept changing between 8.8.8.8 and <some_private_ip> , where <some_private_ip> is the IP address of a private-DNS-server we setup in our network.

This cause an issue as when the current DNS was set to 8.8.8.8, any system-service we ran from the node did not recognize the address we added to our private-DNS-server.

To solve it we hard-coded the <some_private_ip> in /etc/systemd/resolved.conf Under [Resolve] -> DNS:

...
[Resolve]
DNS=<some_private_ip>
...

And ran sudo systemctl restart systemd-resolved.service

However the same issue still exists for the pods inside the cluster.
Running a simple curl to any address we added to our private-DNS-server, sometimes they recognize the host, sometimes they do not.

Help..?