Fedora 31 上部署kubernetes 单机集群，宿主服务器重启后，dashboard不能启动故障分析

柔情只为你懂 2022-12-01 05:16 29阅读 0赞

查看kube-system pods状态

[yeqiang@harbor ~]$ kubectl get pods -n kube-system
    NAME                                         READY   STATUS             RESTARTS   AGE
    coredns-65dbdb44db-xmll6                     0/1     Running            10         5d18h
    dashboard-metrics-scraper-545bbb8767-rx5km   1/1     Running            10         5d18h
    default-http-backend-6c7dd875c-4bm7n         1/1     Running            8          4d16h
    kube-flannel-ds-amd64-7hz8s                  1/1     Running            12         5d18h
    kubernetes-dashboard-65665f84db-hsx8d        0/1     CrashLoopBackOff   23         18h
    metrics-server-869ffc99cd-ppmkf              0/1     CrashLoopBackOff   155        4d17h

其中 kubernetes-dashboard，metrics-server两个pod启动失败，在不断尝试重启。

查看kubernetes-dashboard日志信息

[yeqiang@harbor ~]$ kubectl  logs -n kube-system kubernetes-dashboard-65665f84db-hsx8d
    2020/08/26 01:19:46 Starting overwatch
    2020/08/26 01:19:46 Using namespace: kube-system
    2020/08/26 01:19:46 Using in-cluster config to connect to apiserver
    2020/08/26 01:19:46 Using secret token for csrf signing
    2020/08/26 01:19:46 Initializing csrf token from kubernetes-dashboard-csrf secret
    panic: Get https://10.68.0.1:443/api/v1/namespaces/kube-system/secrets/kubernetes-dashboard-csrf: dial tcp 10.68.0.1:443: connect: no route to host
    
    goroutine 1 [running]:
    github.com/kubernetes/dashboard/src/app/backend/client/csrf.(*csrfTokenManager).init(0xc0005598c0)
    	/home/travis/build/kubernetes/dashboard/src/app/backend/client/csrf/manager.go:41 +0x446
    github.com/kubernetes/dashboard/src/app/backend/client/csrf.NewCsrfTokenManager(...)
    	/home/travis/build/kubernetes/dashboard/src/app/backend/client/csrf/manager.go:66
    github.com/kubernetes/dashboard/src/app/backend/client.(*clientManager).initCSRFKey(0xc00016e080)
    	/home/travis/build/kubernetes/dashboard/src/app/backend/client/manager.go:501 +0xc6
    github.com/kubernetes/dashboard/src/app/backend/client.(*clientManager).init(0xc00016e080)
    	/home/travis/build/kubernetes/dashboard/src/app/backend/client/manager.go:469 +0x47
    github.com/kubernetes/dashboard/src/app/backend/client.NewClientManager(...)
    	/home/travis/build/kubernetes/dashboard/src/app/backend/client/manager.go:550
    main.main()
    	/home/travis/build/kubernetes/dashboard/src/app/backend/dashboard.go:105 +0x20d

可以看到，由于容器网络访问10.68.0.1:443不通导致

查看iptables规则

[yeqiang@harbor ~]$ sudo iptables -L -n
    [sudo] password for yeqiang: 
    Chain INPUT (policy ACCEPT)
    target     prot opt source               destination         
    KUBE-FIREWALL  all  --  0.0.0.0/0            0.0.0.0/0           
    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED,DNAT
    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
    INPUT_direct  all  --  0.0.0.0/0            0.0.0.0/0           
    INPUT_ZONES  all  --  0.0.0.0/0            0.0.0.0/0           
    DROP       all  --  0.0.0.0/0            0.0.0.0/0            ctstate INVALID
    REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited
    
    Chain FORWARD (policy ACCEPT)
    target     prot opt source               destination         
    KUBE-FORWARD  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes forwarding rules */
    DOCKER-USER  all  --  0.0.0.0/0            0.0.0.0/0           
    DOCKER-ISOLATION-STAGE-1  all  --  0.0.0.0/0            0.0.0.0/0           
    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    DOCKER     all  --  0.0.0.0/0            0.0.0.0/0           
    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED,DNAT
    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
    FORWARD_direct  all  --  0.0.0.0/0            0.0.0.0/0           
    FORWARD_IN_ZONES  all  --  0.0.0.0/0            0.0.0.0/0           
    FORWARD_OUT_ZONES  all  --  0.0.0.0/0            0.0.0.0/0           
    DROP       all  --  0.0.0.0/0            0.0.0.0/0            ctstate INVALID
    REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited
    ACCEPT     all  --  172.20.0.0/16        0.0.0.0/0           
    ACCEPT     all  --  0.0.0.0/0            172.20.0.0/16       
    
    Chain OUTPUT (policy ACCEPT)
    target     prot opt source               destination         
    KUBE-FIREWALL  all  --  0.0.0.0/0            0.0.0.0/0           
    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
    OUTPUT_direct  all  --  0.0.0.0/0            0.0.0.0/0           
    
    Chain DOCKER (1 references)
    target     prot opt source               destination         
    
    Chain DOCKER-ISOLATION-STAGE-1 (1 references)
    target     prot opt source               destination         
    DOCKER-ISOLATION-STAGE-2  all  --  0.0.0.0/0            0.0.0.0/0           
    RETURN     all  --  0.0.0.0/0            0.0.0.0/0           
    
    Chain DOCKER-ISOLATION-STAGE-2 (1 references)
    target     prot opt source               destination         
    DROP       all  --  0.0.0.0/0            0.0.0.0/0           
    RETURN     all  --  0.0.0.0/0            0.0.0.0/0           
    
    Chain DOCKER-USER (1 references)
    target     prot opt source               destination         
    RETURN     all  --  0.0.0.0/0            0.0.0.0/0           
    
    Chain FORWARD_IN_ZONES (1 references)
    target     prot opt source               destination         
    FWDI_FedoraWorkstation  all  --  0.0.0.0/0            0.0.0.0/0           [goto] 
    FWDI_FedoraWorkstation  all  --  0.0.0.0/0            0.0.0.0/0           [goto] 
    
    Chain FORWARD_OUT_ZONES (1 references)
    target     prot opt source               destination         
    FWDO_FedoraWorkstation  all  --  0.0.0.0/0            0.0.0.0/0           [goto] 
    FWDO_FedoraWorkstation  all  --  0.0.0.0/0            0.0.0.0/0           [goto] 
    
    Chain FORWARD_direct (1 references)
    target     prot opt source               destination         
    
    Chain FWDI_FedoraWorkstation (2 references)
    target     prot opt source               destination         
    FWDI_FedoraWorkstation_pre  all  --  0.0.0.0/0            0.0.0.0/0           
    FWDI_FedoraWorkstation_log  all  --  0.0.0.0/0            0.0.0.0/0           
    FWDI_FedoraWorkstation_deny  all  --  0.0.0.0/0            0.0.0.0/0           
    FWDI_FedoraWorkstation_allow  all  --  0.0.0.0/0            0.0.0.0/0           
    FWDI_FedoraWorkstation_post  all  --  0.0.0.0/0            0.0.0.0/0           
    ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0           
    
    Chain FWDI_FedoraWorkstation_allow (1 references)
    target     prot opt source               destination         
    
    Chain FWDI_FedoraWorkstation_deny (1 references)
    target     prot opt source               destination         
    
    Chain FWDI_FedoraWorkstation_log (1 references)
    target     prot opt source               destination         
    
    Chain FWDI_FedoraWorkstation_post (1 references)
    target     prot opt source               destination         
    
    Chain FWDI_FedoraWorkstation_pre (1 references)
    target     prot opt source               destination         
    
    Chain FWDO_FedoraWorkstation (2 references)
    target     prot opt source               destination         
    FWDO_FedoraWorkstation_pre  all  --  0.0.0.0/0            0.0.0.0/0           
    FWDO_FedoraWorkstation_log  all  --  0.0.0.0/0            0.0.0.0/0           
    FWDO_FedoraWorkstation_deny  all  --  0.0.0.0/0            0.0.0.0/0           
    FWDO_FedoraWorkstation_allow  all  --  0.0.0.0/0            0.0.0.0/0           
    FWDO_FedoraWorkstation_post  all  --  0.0.0.0/0            0.0.0.0/0           
    
    Chain FWDO_FedoraWorkstation_allow (1 references)
    target     prot opt source               destination         
    
    Chain FWDO_FedoraWorkstation_deny (1 references)
    target     prot opt source               destination         
    
    Chain FWDO_FedoraWorkstation_log (1 references)
    target     prot opt source               destination         
    
    Chain FWDO_FedoraWorkstation_post (1 references)
    target     prot opt source               destination         
    
    Chain FWDO_FedoraWorkstation_pre (1 references)
    target     prot opt source               destination         
    
    Chain INPUT_ZONES (1 references)
    target     prot opt source               destination         
    IN_FedoraWorkstation  all  --  0.0.0.0/0            0.0.0.0/0           [goto] 
    IN_FedoraWorkstation  all  --  0.0.0.0/0            0.0.0.0/0           [goto] 
    
    Chain INPUT_direct (1 references)
    target     prot opt source               destination         
    
    Chain IN_FedoraWorkstation (2 references)
    target     prot opt source               destination         
    IN_FedoraWorkstation_pre  all  --  0.0.0.0/0            0.0.0.0/0           
    IN_FedoraWorkstation_log  all  --  0.0.0.0/0            0.0.0.0/0           
    IN_FedoraWorkstation_deny  all  --  0.0.0.0/0            0.0.0.0/0           
    IN_FedoraWorkstation_allow  all  --  0.0.0.0/0            0.0.0.0/0           
    IN_FedoraWorkstation_post  all  --  0.0.0.0/0            0.0.0.0/0           
    ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0           
    
    Chain IN_FedoraWorkstation_allow (1 references)
    target     prot opt source               destination         
    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:22 ctstate NEW,UNTRACKED
    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:3306 ctstate NEW,UNTRACKED
    
    Chain IN_FedoraWorkstation_deny (1 references)
    target     prot opt source               destination         
    
    Chain IN_FedoraWorkstation_log (1 references)
    target     prot opt source               destination         
    
    Chain IN_FedoraWorkstation_post (1 references)
    target     prot opt source               destination         
    
    Chain IN_FedoraWorkstation_pre (1 references)
    target     prot opt source               destination         
    
    Chain KUBE-FIREWALL (2 references)
    target     prot opt source               destination         
    DROP       all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000
    
    Chain KUBE-FORWARD (1 references)
    target     prot opt source               destination         
    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes forwarding rules */ mark match 0x4000/0x4000
    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes forwarding conntrack pod source rule */ ctstate RELATED,ESTABLISHED
    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes forwarding conntrack pod destination rule */ ctstate RELATED,ESTABLISHED
    
    Chain KUBE-KUBELET-CANARY (0 references)
    target     prot opt source               destination         
    
    Chain OUTPUT_direct (1 references)
    target     prot opt source               destination

**零时处理方案**，清空iptables规则

[yeqiang@harbor ~]$ sudo iptables -F

删除pod，触发k8s立即重新部署新pod

[yeqiang@harbor ~]$ kubectl delete pod -n kube-system kubernetes-dashboard-65665f84db-hsx8d
    pod "kubernetes-dashboard-65665f84db-hsx8d" deleted
    [yeqiang@harbor ~]$ kubectl get pods -n kube-system
    NAME                                         READY   STATUS             RESTARTS   AGE
    coredns-65dbdb44db-xmll6                     1/1     Running            10         5d18h
    dashboard-metrics-scraper-545bbb8767-rx5km   1/1     Running            10         5d18h
    default-http-backend-6c7dd875c-4bm7n         1/1     Running            8          4d16h
    kube-flannel-ds-amd64-7hz8s                  1/1     Running            12         5d18h
    kubernetes-dashboard-65665f84db-wb965        1/1     Running            0          7s
    metrics-server-869ffc99cd-ppmkf              0/1     CrashLoopBackOff   156        4d17h

可以看到kubernetes-dashboard正常启动起来。