I setup a Kubernetes cluster using Kubespray. it works flawlessly until recently I encountered something strange.
I created a BIND DNS deployment using the following manifest.
apiVersion: apps/v1
kind: Deployment
metadata:
name: bind
labels:
app: bind
spec:
replicas: 1
selector:
matchLabels:
app: bind
template:
metadata:
labels:
app: bind
spec:
containers:
- name: bind
image: sameersbn/bind:9.16.1-20200524
ports:
- containerPort: 53
name: tcp
protocol: TCP
- containerPort: 53
name: udp
protocol: UDP
- containerPort: 10000
name: web
protocol: TCP
volumeMounts:
- name: bind-data
mountPath: /data
volumes:
- name: bind-data
hostPath:
path: /data
---
apiVersion: v1
kind: Service
metadata:
name: bind
spec:
type: ClusterIP
ports:
- port: 53
name: tcp
targetPort: tcp
protocol: TCP
- port: 53
name: udp
targetPort: udp
protocol: UDP
- port: 10000
name: web
targetPort: web
protocol: TCP
selector:
app: bind
Then I’ve defined a master zone and created a forward dns record in it.
test.arman.local -> 192.168.230.216
Next I have configured the configmap of coredns to use it:
kubectl edit configmap -n kube-system coredns
apiVersion: v1
data:
Corefile: |
.:53 {
errors
health {
lameduck 5s
}
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
forward . /etc/resolv.conf
cache 30
loop
reload
loadbalance
}
arman.local:53 {
errors
cache 30
forward . 10.233.28.178
}
kind: ConfigMap
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","data":{"Corefile":".:53 {n errors {n }n health {n lameduck 5sn }n readyn kubernetes cluster.local in-addr.arpa ip6.arpa {n pods insecuren fallthrough in-addr.arpa ip6.arpan }n prometheus :9153n forward . /etc/resolv.conf {n prefer_udpn max_concurrent 1000n }n cache 30nn loopn reloadn loadbalancen}n"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"EnsureExists"},"name":"coredns","namespace":"kube-system"}}
creationTimestamp: "2024-08-08T10:26:32Z"
labels:
addonmanager.kubernetes.io/mode: EnsureExists
name: coredns
namespace: kube-system
resourceVersion: "2135167"
uid: 5457c863-3a34-4490-badb-036f4f0e1739
The service of BIND deployment is like:
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
bind ClusterIP 10.233.28.178 <none> 53/TCP,53/UDP,10000/TCP 44h
The problem is it can’t resolve test.arman.local
I’ve started to debug the problem I launched a busybox and used nslookup
kubectl run -it busybox --image=busybox:1.28 --restart=Never -- sh
Here is the content of /etc/resolv.conf of busybox
/ # cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 169.254.25.10
options ndots:5
I don’t know what IP 169.254.25.10 is for because the service of dnsCore has different IP:
kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
coredns ClusterIP 10.233.0.3 <none> 53/UDP,53/TCP,9153/TCP 10d
In the busybox I can successfully resolve the test.arman.local using the following command:
nslookup test.arman.local 10.233.28.178
Server: 10.233.28.178
Address 1: 10.233.28.178
Name: test.arman.local
Address 1: 192.168.230.216
/ # ^C
/ # cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 169.254.25.10
options ndots:5
Or even use this:
nslookup test.arman.local 10.233.0.3 # This is for coredns
Server: 10.233.0.3
Address 1: 10.233.0.3 coredns.kube-system.svc.cluster.local
Name: test.arman.local
Address 1: 192.168.230.216
We are using DNS/calico, for the sake of completeness I post the pods in kube-system:
kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-c7cc688f8-rb59q 1/1 Running 1 (7d21h ago) 10d
calico-node-5dglq 1/1 Running 0 10d
calico-node-c592m 1/1 Running 0 10d
calico-node-hbfqj 1/1 Running 0 10d
calico-node-s6jrl 1/1 Running 0 10d
coredns-857f949bf7-bprlf 1/1 Running 0 65m
coredns-857f949bf7-r9p87 1/1 Running 0 65m
dns-autoscaler-6ffb84bd6-fgdtz 1/1 Running 0 10d
kube-apiserver-node1 1/1 Running 1 10d
kube-apiserver-node2 1/1 Running 1 10d
kube-controller-manager-node1 1/1 Running 4 (7d21h ago) 10d
kube-controller-manager-node2 1/1 Running 7 (6d19h ago) 10d
kube-proxy-hb2wn 1/1 Running 0 10d
kube-proxy-j5pfg 1/1 Running 0 10d
kube-proxy-qjj4n 1/1 Running 0 10d
kube-proxy-rp9mt 1/1 Running 0 10d
kube-scheduler-node1 1/1 Running 4 (6d19h ago) 10d
kube-scheduler-node2 1/1 Running 6 (6d19h ago) 10d
nginx-proxy-node3 1/1 Running 0 10d
nginx-proxy-node4 1/1 Running 0 10d
nodelocaldns-6sztv 1/1 Running 0 10d
nodelocaldns-bpt7s 1/1 Running 0 10d
nodelocaldns-n4rsk 1/1 Running 0 10d
nodelocaldns-n8q9w 1/1 Running 0 10d