kubernetes的dnsmasq缓存查询结果,导致pod偶尔无法访问域名

Tags: kubernetes_problem 

目录

现象

在kubernete的pod中,访问域名repo1.maven.org的时候,发现无法访问:

$ping repo1.maven.org
 ... Bad address ...

pod的使用的域名服务器是kubernetes中部署的kube-dns。

$cat /etc/resolv.conf
search admin.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.0.0.10
options ndots:5

重启kube-dns的pod后,问题消失,但是经过不确定的时间后,问题又出现。

开启dnsmasq的日志

kube-dns中直接对外提供服务的是运行dnsmasq的容器,在kube-dns.yaml中可以看到:

- args:
        - --cache-size=1000
        - --no-resolv
        - --server=/cluster.local/127.0.0.1#10053
        - --server=10.36.8.40
        - --server=10.36.8.41
        image: XXXXX/kube-dnsmasq-amd64:1.3

要打开日志功能,需要将其修改为:

- args:
        - -q
        - --cache-size=1000
        - --no-resolv
        - --log-facility=/dev/stdout
        - --server=/cluster.local/127.0.0.1#10053
        - --server=10.36.8.40
        - --server=10.36.8.41
        image: XXXXX/kube-dnsmasq-amd64:1.3

参数-q表示记录查询日志,--log-facility指定日志文件。

分析日志

第一次域名解析,以及成功的时候

Aug  3 02:11:35 dnsmasq[1]: query[A] repo1.maven.org from 192.168.91.1\n","stream":"stdout","time":"2017-08-03T02:11:35.696997573Z
Aug  3 02:11:35 dnsmasq[1]: forwarded repo1.maven.org to 10.36.8.41\n","stream":"stdout","time":"2017-08-03T02:11:35.697008546Z
Aug  3 02:11:35 dnsmasq[1]: query[A] repo1.maven.org from 192.168.91.1\n","stream":"stdout","time":"2017-08-03T02:11:35.697011973Z
Aug  3 02:11:35 dnsmasq[1]: forwarded repo1.maven.org to 10.36.8.41\n","stream":"stdout","time":"2017-08-03T02:11:35.697181313Z
...
Aug  3 02:11:35 dnsmasq[1]: reply repo1.maven.org is \u003cCNAME\u003e\n","stream":"stdout","time":"2017-08-03T02:11:35.822215656Z
Aug  3 02:11:35 dnsmasq[1]: reply central.maven.org is \u003cCNAME\u003e\n","stream":"stdout","time":"2017-08-03T02:11:35.822232258Z
Aug  3 02:11:35 dnsmasq[1]: reply sonatype.map.fastly.net is 151.101.40.209\n","stream":"stdout","time":"2017-08-03T02:11:35.822236436Z
Aug  3 02:11:35 dnsmasq[1]: reply repo1.maven.org is \u003cCNAME\u003e\n","stream":"stdout","time":"2017-08-03T02:11:35.822239784Z
Aug  3 02:11:35 dnsmasq[1]: reply central.maven.org is \u003cCNAME\u003e\n","stream":"stdout","time":"2017-08-03T02:11:35.822243195Z
Aug  3 02:11:35 dnsmasq[1]: reply sonatype.map.fastly.net is 151.101.40.209\n","stream":"stdout","time":"2017-08-03T02:11:35.822246586Z
...
Aug  3 02:11:36 dnsmasq[1]: query[A] repo1.maven.org from 192.168.91.1\n","stream":"stdout","time":"2017-08-03T02:11:36.031788497Z
Aug  3 02:11:36 dnsmasq[1]: cached repo1.maven.org is \u003cCNAME\u003e\n","stream":"stdout","time":"2017-08-03T02:11:36.031803088Z
Aug  3 02:11:36 dnsmasq[1]: cached central.maven.org is \u003cCNAME\u003e\n","stream":"stdout","time":"2017-08-03T02:11:36.031807428Z
Aug  3 02:11:36 dnsmasq[1]: cached sonatype.map.fastly.net is 151.101.40.209\n","stream":"stdout","time":"2017-08-03T02:11:36.031811367Z

可以看到,dnsmasq将repo1.maven.org转发到上游的DNS Server,并且最终得到上游DNS Server的返回的IP。

dnsmasq缓存了上游DNS Server返回的结果,在后续的查询请求中,直接返回cache的结果。

解析失败的时候

Aug  3 02:27:15 dnsmasq[1]: query[A] repo1.maven.org from 192.168.91.1\n","stream":"stdout","time":"2017-08-03T02:27:15.462049449Z
Aug  3 02:27:15 dnsmasq[1]: forwarded repo1.maven.org to 10.36.8.40\n","stream":"stdout","time":"2017-08-03T02:27:15.462052408Z
Aug  3 02:27:15 dnsmasq[1]: forwarded repo1.maven.org to 10.36.8.41\n","stream":"stdout","time":"2017-08-03T02:27:15.462058375Z
...
Aug  3 02:27:17 dnsmasq[1]: reply repo1.maven.org is \u003cCNAME\u003e\n","stream":"stdout","time":"2017-08-03T02:27:17.92674093Z
Aug  3 02:27:17 dnsmasq[1]: reply central.maven.org is \u003cCNAME\u003e\n","stream":"stdout","time":"2017-08-03T02:27:17.926778956Z
Aug  3 02:27:17 dnsmasq[1]: reply sonatype.map.fastly.net is NODATA-IPv4\n","stream":"stdout","time":"2017-08-03T02:27:17.926783393Z
Aug  3 02:27:17 dnsmasq[1]: reply repo1.maven.org is \u003cCNAME\u003e\n","stream":"stdout","time":"2017-08-03T02:27:17.926786834Z
...
Aug  3 02:27:19 dnsmasq[1]: query[A] repo1.maven.org from 192.168.91.1\n","stream":"stdout","time":"2017-08-03T02:27:19.659929868Z
Aug  3 02:27:19 dnsmasq[1]: cached repo1.maven.org is \u003cCNAME\u003e\n","stream":"stdout","time":"2017-08-03T02:27:19.659945829Z
Aug  3 02:27:19 dnsmasq[1]: cached central.maven.org is \u003cCNAME\u003e\n","stream":"stdout","time":"2017-08-03T02:27:19.659950352Z
Aug  3 02:27:19 dnsmasq[1]: cached sonatype.map.fastly.net is NODATA-IPv4\n","stream":"stdout","time":"2017-08-03T02:27:19.659954325Z

域名被转发到上游的DNS Server,但是上游的DNS Server返回的是NODATA-IPv4

dnsmasq缓存了这个结果,在后续的查询中,也返回NODATA-IPv4

分析域名repo1.maven.org

得到域名repo1.maven.org的IP地址,需要经过下面的过程:

Query:        repo1.maven.org
Query cname:  central.maven.org
Query cname:  sonatype.map.fastly.net
GET IP:       151.101.40.209

用dig不停地查询域名sonatype.map.fastly.net,发现这个域名的查询结果有以下几种:

查询到了A记录:

$ dig @10.36.8.40 sonatype.map.fastly.net

; <<>> DiG 9.8.3-P1 <<>> sonatype.map.fastly.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 22582
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;sonatype.map.fastly.net.	IN	A

;; ANSWER SECTION:
sonatype.map.fastly.net. 26	IN	A	151.101.24.209

;; Query time: 3699 msec
;; SERVER: 10.36.8.41#53(10.36.8.41)
;; WHEN: Thu Aug  3 14:53:33 2017
;; MSG SIZE  rcvd: 57

没有查询到:

dig @10.36.8.40  sonatype.map.fastly.net

; <<>> DiG 9.8.3-P1 <<>> @10.36.8.40 sonatype.map.fastly.net
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 44393
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;sonatype.map.fastly.net.	IN	A

;; Query time: 2553 msec
;; SERVER: 10.36.8.40#53(10.36.8.40)
;; WHEN: Thu Aug  3 14:57:53 2017
;; MSG SIZE  rcvd: 41

由此可知,上游的DNS Server不能稳定的响应对该域名的查询。

解决方法

根本原因在于上游的DNS Server,当上游DNS Server没有返回正确的结果时,dnsmasq将查询失败的结果缓存,导致dnsmasq的用户长时间得不到正确的查询结果。

好在dnsmasq还有这样一个配置项:

-N, --no-negcache                       Do NOT cache failed search results.

编辑kube-dns,将其开启:

- args:
        - -N
        - -q
        - --cache-size=1000
        - --no-resolv
        - --log-facility=/dev/stdout
        - --server=/cluster.local/127.0.0.1#10053
        - --server=10.36.8.40
        - --server=10.36.8.41
        image: XXXXX/kube-dnsmasq-amd64:1.3

参考

  1. NODATA-IPv4

kubernetes_problem

  1. kubernetes ingress-nginx 启用 upstream 长连接,需要注意,否则容易 502
  2. kubernetes ingress-nginx 的 canary 影响指向同一个 service 的所有 ingress
  3. ingress-nginx 启用 tls 加密,配置了不存在的证书,导致 unable to get local issuer certificate
  4. https 协议访问,误用 http 端口,CONNECT_CR_SRVR_HELLO: wrong version number
  5. Kubernetes ingress-nginx 4 层 tcp 代理,无限重试不存在的地址,高达百万次
  6. Kubernetes 集群中个别 Pod 的 CPU 使用率异常高的问题调查
  7. Kubernetes 集群 Node 间歇性变为 NotReady 状态: IO 负载高,延迟严重
  8. Kubernetes的nginx-ingress-controller刷新nginx的配置滞后十分钟导致504
  9. Kubernetes的Nginx Ingress 0.20之前的版本,upstream的keep-alive不生效
  10. Kubernetes node 的 xfs文件系统损坏,kubelet主动退出且重启失败,恢复后无法创建pod
  11. Kubernetes的Pod无法删除,glusterfs导致docker无响应,集群雪崩
  12. Kubernetes集群node无法访问service: kube-proxy没有正确设置cluster-cidr
  13. Kubernetes集群node上的容器无法ping通外网: iptables snat规则缺失导致
  14. Kubernetes问题调查: failed to get cgroup stats for /systemd/system.slice
  15. Kubelet1.7.16使用kubeconfig时,没有设置--require-kubeconfig,导致node不能注册
  16. Kubelet从1.7.16升级到1.9.11,Sandbox以外的容器都被重建的问题调查
  17. Kubernetes: 内核参数rp_filter设置为Strict RPF,导致Service不通
  18. Kubernetes使用过程中遇到的一些问题与解决方法
  19. Kubernetes集群节点被入侵挖矿,CPU被占满
  20. kubernetes的node上的重启linux网络服务后,pod无法联通
  21. kubernetes的pod因为同名Sandbox的存在,一直无法删除
  22. kubelet升级,导致calico中存在多余的workloadendpoint,node上存在多余的veth设备
  23. 使用petset创建的etcd集群在kubernetes中运行失败
  24. Kubernetes 容器启动失败: unable to create nf_conn slab cache
  25. 未在calico中创建hostendpoint,导致开启隔离后,在kubernetes的node上无法访问pod
  26. calico分配的ip冲突,pod内部arp记录丢失,pod无法访问外部服务
  27. kubernetes的dnsmasq缓存查询结果,导致pod偶尔无法访问域名
  28. k8s: rbd image is locked by other nodes
  29. kuberntes的node无法通过物理机网卡访问Service

推荐阅读

Copyright @2011-2019 All rights reserved. 转载请添加原文连接,合作请加微信lijiaocn或者发送邮件: [email protected],备注网站合作

友情链接:  李佶澳的博客  系统软件  程序语言  运营经验  关注方向  水库文集  网文收藏  网络课程  发现知识星球  百度搜索 谷歌搜索