coredns service 连接超时(connection timed out; no servers could be reached)
node 主机测试kube-dns 解析提示连接超时
dig @10.96.0.10 kube-dns.kube-system.svc.cluster.local
# 显示如下
; <<>> DiG 9.11.26-RedHat-9.11.26-6.el8 <<>> @10.96.0.10 kube-dns.kube-system.svc.cluster.local
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached修改 coredns configmap 配置，查看coredns 日志
kubectl -n kube-system edit cm coredns在 Corefile 增加log 配置
apiVersion: v1
data:
Corefile: |
.:53 {
log
errors
health {
lameduck 5s
}
...保存更改后，k8s 将这些更改传播到CoreDNS豆荚可能需要一到两分钟的时间。CoreDNS会自动加载配置。
查询coredns 所有pod日志
kubectl logs coredns-5f9bcf7c57-hbq5z -n kube-system --tail 100显示类似如下内容，没有异常信息
[INFO] 10.128.8.110:40481 - 26090 "A IN kafka-zk-zookeeper-headless.zookeeper.svc.cluster.local.svc.cluster.local. udp 91 false 512" NXDOMAIN qr,aa,rd 184 0.000058542s
[INFO] 10.128.8.110:33032 - 61678 "A IN kafka-zk-zookeeper-headless.zookeeper.svc.cluster.local. udp 73 false 512" NOERROR qr,aa,rd 144 0.000067122s
[INFO] 10.128.3.156:54546 - 33121 "A IN dev-mgo-nft-mongodb.nft.svc.nft.svc.cluster.local. udp 67 false 512" NXDOMAIN qr,aa,rd 160 0.000404809s
[INFO] 10.128.3.156:35509 - 17309 "A IN dev-mgo-nft-mongodb.nft.svc.svc.cluster.local. udp 63 false 512" NXDOMAIN qr,aa,rd 156 0.000294607s
[INFO] 10.128.3.156:41908 - 8575 "AAAA IN dev-mgo-nft-mongodb.nft.svc.cluster.local. udp 59 false 512" NOERROR qr,aa,rd 152 0.000117004s
[INFO] 10.128.8.110:59090 - 31404 "AAAA IN kafka-zk-zookeeper-headless.zookeeper.svc.cluster.local.svc.cluster.local. udp 91 false 512" NXDOMAIN qr,aa,rd 184 0.000143273s
[INFO] 10.128.8.110:45311 - 21795 "AAAA IN kafka-zk-zookeeper-headless.zookeeper.svc.cluster.local. udp 73 false 512" NOERROR qr,aa,rd 166 0.000079302s
[INFO] 10.128.8.110:42297 - 55594 "A IN kafka-zk-zookeeper-headless.zookeeper.svc.cluster.local.svc.cluster.local. udp 91 false 512" NXDOMAIN qr,aa,rd 184 0.000044621s
[INFO] 10.128.8.110:58692 - 11604 "A IN kafka-zk-zookeeper-headless.zookeeper.svc.cluster.local. udp 73 false 512" NOERROR qr,aa,rd 144 0.000041031s查看 svc 和 endpoints信息
# kubectl -n kube-system get endpoints kube-dns
NAME ENDPOINTS AGE
kube-dns 10.128.0.58:53,10.128.6.152:53,10.128.0.58:53 + 3 more... 15h
[root@master1 ~]# kubectl -n kube-system describe endpoints kube-dns
Name: kube-dns
Namespace: kube-system
Labels: k8s-app=kube-dns
kubernetes.io/cluster-service=true
kubernetes.io/name=CoreDNS
Annotations: endpoints.kubernetes.io/last-change-trigger-time: 2022-11-24T10:57:23Z
Subsets:
Addresses: 10.128.0.58,10.128.6.152
NotReadyAddresses: <none>
Ports:
Name Port Protocol
---- ---- --------
dns-tcp 53 TCP
dns 53 UDP
metrics 9153 TCP
Events: <none>使用dig 测试 pod ip ,查看解析
dig @10.128.0.58 kube-dns.kube-system.svc.cluster.local
dig @10.128.6.152 kube-dns.kube-system.svc.cluster.local结果都是正常的, 说明coredns 服务本身没有问题
; <<>> DiG 9.11.26-RedHat-9.11.26-6.el8 <<>> @10.128.0.58 kube-dns.kube-system.svc.cluster.local
; (1 server found)
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41299
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 75dd2e6ddd4a4454 (echoed)
;; QUESTION SECTION:
;kube-dns.kube-system.svc.cluster.local. IN A
;; ANSWER SECTION:
kube-dns.kube-system.svc.cluster.local. 30 IN A 10.96.0.10
;; Query time: 0 msec
;; SERVER: 10.128.0.58#53(10.128.0.58)
;; WHEN: Fri Nov 25 10:18:28 CST 2022
;; MSG SIZE rcvd: 133telnet 测试连通性
使用telnet 在master 主机和 node 测试 10.96.0.10 53 端口，结果都是正常的
telnet 10.96.0.10 53
# 显示如下
Trying 10.96.0.10...
Connected to 10.96.0.10.
Escape character is '^]'.
Connection closed by foreign host.根据网上搜索相关错误联系刚升级的flannel服务配置解决问题
根据 https://github.com/coredns/coredns/issues/3704 中信息提示，pod 网络和服务网络重叠会导致路由问题。
我猜测 可能跟我 刚更新的 cni flannel 版本到v0.20.1 有关系
flannel Network 配置如下
net-conf.json: |
{
"Network": "10.128.0.0/16",
"Backend": {
"Type": "vxlan"
}
}Kubeadm-config 中Networking配置如下
networking:
dnsDomain: cluster.local
serviceSubnet: 10.96.0.0/12
podSubnet: 10.128.0.0/16Kubeadm-config 中 serviceSubnet 不在 flannel Network 配置范围中，我修改了 flannel Network 配置如下
net-conf.json: |
{
"Network": "10.0.0.0/8",
"Backend": {
"Type": "vxlan"
}
}重启flannel ds
kubectl rollout restart ds kube-flannel-ds -n kube-flannel重新 使用dig 测试 kube-dns 域名
dig @10.96.0.10 kube-dns.kube-system.svc.cluster.local可以正常解析
; <<>> DiG 9.11.26-RedHat-9.11.26-6.el8 <<>> @10.96.0.10 kube-dns.kube-system.svc.cluster.local
; (1 server found)
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20581
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: a7918abdce60333a (echoed)
;; QUESTION SECTION:
;kube-dns.kube-system.svc.cluster.local. IN A
;; ANSWER SECTION:
kube-dns.kube-system.svc.cluster.local. 30 IN A 10.96.0.10
;; Query time: 0 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Fri Nov 25 10:33:40 CST 2022
;; MSG SIZE rcvd: 133参考：
https://medium.com/geekculture/k8s-troubleshooting-how-to-debug-coredns-issues-724e8b973cfc
https://github.com/coredns/coredns/issues/3704
https://coredns.io/manual/configuration/
https://kubernetes.io/zh-cn/docs/tasks/administer-cluster/dns-custom-nameservers/
https://github.com/kubernetes-sigs/kubespray/issues/4674
https://github.com/coredns/deployment/blob/master/kubernetes/Upgrading_CoreDNS.md
https://github.com/flannel-io/flannel/blob/v0.20.1/Documentation/configuration.md
Last updated