【kubernetes】k8s的安装详细说明【创建集群、加入集群、踢出集群、重置集群...】

本是古典何须时尚 2022-10-11 13:57 337阅读 0赞

### 文章目录 ###

*  说明
 *  虚拟机环境配置【3台主机同步】
 *   *  配置selinux
     *  配置防火墙
     *  配置解析
     *  关闭swap
     *  配置yum源
     *  安装docker并启动服务
     *  配置加速器
     *  设置内核参数
     *  安装k8s
     *   *  查看可用版本
         *  安装
         *  启动k8s服务
 *  安装kubeadmin【创建集群】【master上操作】
 *   *  安装kubeadmin
     *   *  命令说明
         *  命令执行后镜像一览
         *  命令执行报错及处理
     *  启动kubeadmin
     *  查看当前集群信息
 *  加入集群【node节点上操作】
 *   *  加入集群【kubeadmin】
 *  网络插件calico安装
 *   *  calico.yam文件获取
     *  查看calico所需镜像并下载【3台主机同步】
     *  安装calico【master上操作】
 *  kubectl子命令tab设置
 *  删除集群节点【踢出集群】【master上操作】
 *  集群初始化【master上操作】
 *  node节点重新加入集群操作【node节点操作】
 *   *  加入集群报错原因及内容
     *  处理方法

# 说明 #

*  k8s最好准备3台机子，其中：
    
     *  一台master节点
     *  2台node节点
 *  我环境中的3台主机名和ip分别如下：  
    注：主机名不要使用localhost.localdomain
    
     *  maset：192.168.59.142
     *  node1：192.168.59.143
     *  node2：192.168.59.144
 *  这些主机不一定非要通外网才能做啊，我的这3台主机是没有通外网的，但我准备了一台通外网的主机，将所有需要安装的包通过离线的方式上传到这3服务器上的，方法见下面这篇博客。  
    [【yum】linux离线安装rpm包及其依赖和卸载包及其依赖][yum_linux_rpm]

# 虚拟机环境配置【3台主机同步】 #

注：3台主机【master和node节点】需要做相同操作啊

## 配置selinux ##

防火墙一定要为`disable`模式。  
修改下面配置文件中`SELINUX=disabled`，然后重启生效。

[root@master ~]# cat /etc/selinux/config | grep SELINUX=disa
    SELINUX=disabled
    [root@master ~]# 
    [root@master ~]# getenforce 
    Disabled
    [root@master ~]#

## 配置防火墙 ##

*  如果防火墙没有特别需求，则直接关闭：`systemctl stop firewalld.service`
 *  如果防火墙不能关闭，则设置为默认zone即可：`firewall-cmd --set-default-zone=trusted`  
    我下面所有环境开启防火墙设置默认zone为例

[root@master ~]# systemctl is-active firewalld
    active
    [root@master ~]# firewall-cmd --set-default-zone=trusted
    success
    [root@master ~]#

## 配置解析 ##

3个ip和主机名都添加到解析文件中，然后可以通过scp拷贝到另外两个主机上。

[root@master ~]# tail -n 4 /etc/hosts
    192.168.59.142 master
    192.168.59.143 node1
    192.168.59.144 node2
    
    [root@master ~]#
    [root@master ~]# scp /etc/hosts node1:/etc/hosts
    The authenticity of host 'node1 (192.168.59.143)' can't be established. ECDSA key fingerprint is SHA256:+JrT4G9aMhaod/a9gBjUOzX5aONqQ7a4OX0Oj3Z978c. ECDSA key fingerprint is MD5:7f:4c:cc:5c:10:d2:54:d8:3c:dd:da:39:48:30:12:59. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'node1,192.168.59.143' (ECDSA) to the list of known hosts. root@node1's password: 
    Permission denied, please try again.
    root@node1's password: hosts 100% 224 145.7KB/s 00:00 [root@master ~]# scp /etc/hosts node2:/etc/hosts The authenticity of host 'node2 (192.168.59.144)' can't be established.
    ECDSA key fingerprint is SHA256:+JrT4G9aMhaod/a9gBjUOzX5aONqQ7a4OX0Oj3Z978c.
    ECDSA key fingerprint is MD5:7f:4c:cc:5c:10:d2:54:d8:3c:dd:da:39:48:30:12:59.
    Are you sure you want to continue connecting (yes/no)? yes
    Warning: Permanently added 'node2,192.168.59.144' (ECDSA) to the list of known hosts.
    root@node2's password: 
    hosts                                                                                100%  224   169.6KB/s   00:00    
    [root@master ~]#

## 关闭swap ##

*  命令：`swapoff -a ; sed -i '/swap/d' /etc/fstab`  
    上面是两条命令，前面是临时关闭，后面是关闭配置文件，使重启后也是关闭的。

[root@master ~]# swapoff -a ; sed -i '/swap/d' /etc/fstab
    [root@master ~]#

配置文件中一定要修改，否则机子重启后，k8s会起不来。

## 配置yum源 ##

*  先进入/etc/yum.repos.d/目录，新建一个文件夹，将现在的全部移到文件夹中【删除也行】
 *  然后下载docker的网络源到上面目录中：`wget ftp://ftp.rhce.cc/k8s/* -P /etc/yum.repos.d/`  
    【ftp://ftp.rhce.cc/k8s/\*这是阿里云的镜像地址】

[root@master ~]# 
    [root@master ~]# cd /etc/yum.repos.d/
    [root@master yum.repos.d]# mkdir bak
    [root@master yum.repos.d]# mv * bak
    [root@master yum.repos.d]# wget ftp://ftp.rhce.cc/k8s/* -P /etc/yum.repos.d/
    --2021-06-16 16:27:53--  ftp://ftp.rhce.cc/k8s/*
               => ‘/etc/yum.repos.d/.listing’
    Resolving ftp.rhce.cc (ftp.rhce.cc)... 101.37.152.41
    Connecting to ftp.rhce.cc (ftp.rhce.cc)|101.37.152.41|:21... connected.
    Logging in as anonymous ... Logged in!
    ==> SYST ... done.    ==> PWD ... done.
    ==> TYPE I ... done.  ==> CWD (1) /k8s ... done.
    ==> PASV ... done.    ==> LIST ... done.
    
        [ <=>                                                                  ] 398         --.-K/s   in 0s      
    
    2021-06-16 16:27:55 (45.7 MB/s) - ‘/etc/yum.repos.d/.listing’ saved [398]
    
    Removed ‘/etc/yum.repos.d/.listing’.
    --2021-06-16 16:27:55--  ftp://ftp.rhce.cc/k8s/CentOS-Base.repo
               => ‘/etc/yum.repos.d/CentOS-Base.repo’
    ==> CWD not required.
    ==> PASV ... done.    ==> RETR CentOS-Base.repo ... done.
    Length: 2206 (2.2K)
    
    100%[=====================================================================>] 2,206       --.-K/s   in 0s      
    
    2021-06-16 16:27:56 (162 MB/s) - ‘/etc/yum.repos.d/CentOS-Base.repo’ saved [2206]
    
    --2021-06-16 16:27:56--  ftp://ftp.rhce.cc/k8s/docker-ce.repo
               => ‘/etc/yum.repos.d/docker-ce.repo’
    ==> CWD not required.
    ==> PASV ... done.    ==> RETR docker-ce.repo ... done.
    Length: 2640 (2.6K)
    
    100%[=====================================================================>] 2,640       --.-K/s   in 0s      
    
    2021-06-16 16:27:56 (127 MB/s) - ‘/etc/yum.repos.d/docker-ce.repo’ saved [2640]
    
    --2021-06-16 16:27:56--  ftp://ftp.rhce.cc/k8s/epel.repo
               => ‘/etc/yum.repos.d/epel.repo’
    ==> CWD not required.
    ==> PASV ... done.    ==> RETR epel.repo ... done.
    Length: 923
    
    100%[=====================================================================>] 923         --.-K/s   in 0.006s  
    
    2021-06-16 16:27:56 (163 KB/s) - ‘/etc/yum.repos.d/epel.repo’ saved [923]
    
    --2021-06-16 16:27:56--  ftp://ftp.rhce.cc/k8s/k8s.repo
               => ‘/etc/yum.repos.d/k8s.repo’
    ==> CWD not required.
    ==> PASV ... done.    ==> RETR k8s.repo ... done.
    Length: 276
    
    100%[=====================================================================>] 276         --.-K/s   in 0s      
    
    2021-06-16 16:27:57 (24.3 MB/s) - ‘/etc/yum.repos.d/k8s.repo’ saved [276]
    
    [root@master yum.repos.d]#

## 安装docker并启动服务 ##

k8s是需要依赖运行时的，就是这个`runtime`，我们使用docker的就行。

*  `yum -y install docker-ce` ： 安装docker服务【一共有9个包】
 *  `systemctl enable docker --now`：开机自启并现在启动
 *  `docker info` \# 查看详细信息

[root@master yum.repos.d]# yum -y install docker-ce
    Loaded plugins: fastestmirror, langpacks
    Determining fastest mirrors
    base                                                                           | 3.6 kB  00:00:00     
    docker-ce-stable                                                               | 3.5 kB  00:00:00     
    epel                                                                           | 4.7 kB  00:00:00     
    extras                                                                         | 2.9 kB  00:00:00     
    kubernetes/signature                                                           |  844 B  00:00:00     
    Retrieving key from https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
    Importing GPG key 0x307EA071:
     Userid     : "Rapture Automatic Signing Key (cloud-rapture-signing-key-2021-03-01-08_01_09.pub)"
     Fingerprint: 7f92 e05b 3109 3bef 5a3c 2d38 feea 9169 307e a071
     From       : https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
    Retrieving key from https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
    kubernetes/signature                                                           | 1.4 kB  00:00:00 !!! 
    updates                                                                        | 2.9 kB  00:00:00     
    (1/10): epel/x86_64/group_gz                                                   |  96 kB  00:00:00     
    (2/10): docker-ce-stable/x86_64/updateinfo                                     |   55 B  00:00:00     
    (3/10): docker-ce-stable/x86_64/primary_db                                     |  62 kB  00:00:00     
    (4/10): base/7/x86_64/group_gz                                                 | 153 kB  00:00:00     
    (5/10): epel/x86_64/updateinfo                                                 | 1.0 MB  00:00:00     
    (6/10): kubernetes/primary                                                     |  92 kB  00:00:00     
    (7/10): extras/7/x86_64/primary_db                                             | 242 kB  00:00:00     
    (8/10): base/7/x86_64/primary_db                                               | 6.1 MB  00:00:01     
    (9/10): epel/x86_64/primary_db                                                 | 6.9 MB  00:00:02     
    (10/10): updates/7/x86_64/primary_db                                           | 8.8 MB  00:00:02     
    kubernetes                                                                                    678/678
    Resolving Dependencies
    --> Running transaction check
    ---> Package docker-ce.x86_64 3:20.10.7-3.el7 will be installed
    --> Processing Dependency: container-selinux >= 2:2.74 for package: 3:docker-ce-20.10.7-3.el7.x86_64
    --> Processing Dependency: containerd.io >= 1.4.1 for package: 3:docker-ce-20.10.7-3.el7.x86_64
    --> Processing Dependency: docker-ce-cli for package: 3:docker-ce-20.10.7-3.el7.x86_64
    --> Processing Dependency: docker-ce-rootless-extras for package: 3:docker-ce-20.10.7-3.el7.x86_64
    --> Running transaction check
    ---> Package container-selinux.noarch 2:2.119.2-1.911c772.el7_8 will be installed
    ---> Package containerd.io.x86_64 0:1.4.6-3.1.el7 will be installed
    ---> Package docker-ce-cli.x86_64 1:20.10.7-3.el7 will be installed
    --> Processing Dependency: docker-scan-plugin(x86-64) for package: 1:docker-ce-cli-20.10.7-3.el7.x86_64
    ---> Package docker-ce-rootless-extras.x86_64 0:20.10.7-3.el7 will be installed
    --> Processing Dependency: fuse-overlayfs >= 0.7 for package: docker-ce-rootless-extras-20.10.7-3.el7.x86_64
    --> Processing Dependency: slirp4netns >= 0.4 for package: docker-ce-rootless-extras-20.10.7-3.el7.x86_64
    --> Running transaction check
    ---> Package docker-scan-plugin.x86_64 0:0.8.0-3.el7 will be installed
    ---> Package fuse-overlayfs.x86_64 0:0.7.2-6.el7_8 will be installed
    --> Processing Dependency: libfuse3.so.3(FUSE_3.2)(64bit) for package: fuse-overlayfs-0.7.2-6.el7_8.x86_64
    --> Processing Dependency: libfuse3.so.3(FUSE_3.0)(64bit) for package: fuse-overlayfs-0.7.2-6.el7_8.x86_64
    --> Processing Dependency: libfuse3.so.3()(64bit) for package: fuse-overlayfs-0.7.2-6.el7_8.x86_64
    ---> Package slirp4netns.x86_64 0:0.4.3-4.el7_8 will be installed
    --> Running transaction check
    ---> Package fuse3-libs.x86_64 0:3.6.1-4.el7 will be installed
    --> Finished Dependency Resolution
    
    Dependencies Resolved
    
    ======================================================================================================
     Package                       Arch       Version                          Repository            Size
    ======================================================================================================
    Installing:
     docker-ce                     x86_64     3:20.10.7-3.el7                  docker-ce-stable      27 M
    Installing for dependencies:
     container-selinux             noarch     2:2.119.2-1.911c772.el7_8        extras                40 k
     containerd.io                 x86_64     1.4.6-3.1.el7                    docker-ce-stable      34 M
     docker-ce-cli                 x86_64     1:20.10.7-3.el7                  docker-ce-stable      33 M
     docker-ce-rootless-extras     x86_64     20.10.7-3.el7                    docker-ce-stable     9.2 M
     docker-scan-plugin            x86_64     0.8.0-3.el7                      docker-ce-stable     4.2 M
     fuse-overlayfs                x86_64     0.7.2-6.el7_8                    extras                54 k
     fuse3-libs                    x86_64     3.6.1-4.el7                      extras                82 k
     slirp4netns                   x86_64     0.4.3-4.el7_8                    extras                81 k
    
    Transaction Summary
    ======================================================================================================
    Install  1 Package (+8 Dependent packages)
    
    Total download size: 107 M
    Installed size: 438 M
    Downloading packages:
    warning: /var/cache/yum/x86_64/7/extras/packages/container-selinux-2.119.2-1.911c772.el7_8.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
    Public key for container-selinux-2.119.2-1.911c772.el7_8.noarch.rpm is not installed
    (1/9): container-selinux-2.119.2-1.911c772.el7_8.noarch.rpm                    |  40 kB  00:00:00     
    warning: /var/cache/yum/x86_64/7/docker-ce-stable/packages/docker-ce-20.10.7-3.el7.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 621e9f35: NOKEY
    Public key for docker-ce-20.10.7-3.el7.x86_64.rpm is not installed
    (2/9): docker-ce-20.10.7-3.el7.x86_64.rpm                                      |  27 MB  00:00:06     
    (3/9): containerd.io-1.4.6-3.1.el7.x86_64.rpm                                  |  34 MB  00:00:08     
    (4/9): docker-ce-rootless-extras-20.10.7-3.el7.x86_64.rpm                      | 9.2 MB  00:00:02     
    (5/9): fuse-overlayfs-0.7.2-6.el7_8.x86_64.rpm                                 |  54 kB  00:00:00     
    (6/9): slirp4netns-0.4.3-4.el7_8.x86_64.rpm                                    |  81 kB  00:00:00     
    (7/9): fuse3-libs-3.6.1-4.el7.x86_64.rpm                                       |  82 kB  00:00:00     
    (8/9): docker-scan-plugin-0.8.0-3.el7.x86_64.rpm                               | 4.2 MB  00:00:01     
    (9/9): docker-ce-cli-20.10.7-3.el7.x86_64.rpm                                  |  33 MB  00:00:08     
    ------------------------------------------------------------------------------------------------------
    Total                                                                 7.1 MB/s | 107 MB  00:00:15     
    Retrieving key from https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
    Importing GPG key 0x621E9F35:
     Userid     : "Docker Release (CE rpm) <docker@docker.com>"
     Fingerprint: 060a 61c5 1b55 8a7f 742b 77aa c52f eb6b 621e 9f35
     From       : https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
    Retrieving key from http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7
    Importing GPG key 0xF4A80EB5:
     Userid     : "CentOS-7 Key (CentOS 7 Official Signing Key) <security@centos.org>"
     Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
     From       : http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7
    Running transaction check
    Running transaction test
    Transaction test succeeded
    Running transaction
      Installing : 2:container-selinux-2.119.2-1.911c772.el7_8.noarch                                 1/9 
      Installing : containerd.io-1.4.6-3.1.el7.x86_64                                                 2/9 
      Installing : 1:docker-ce-cli-20.10.7-3.el7.x86_64                                               3/9 
      Installing : docker-scan-plugin-0.8.0-3.el7.x86_64                                              4/9 
      Installing : slirp4netns-0.4.3-4.el7_8.x86_64                                                   5/9 
      Installing : fuse3-libs-3.6.1-4.el7.x86_64                                                      6/9 
      Installing : fuse-overlayfs-0.7.2-6.el7_8.x86_64                                                7/9 
      Installing : docker-ce-rootless-extras-20.10.7-3.el7.x86_64                                     8/9 
      Installing : 3:docker-ce-20.10.7-3.el7.x86_64                                                   9/9 
      Verifying  : containerd.io-1.4.6-3.1.el7.x86_64                                                 1/9 
      Verifying  : fuse3-libs-3.6.1-4.el7.x86_64                                                      2/9 
      Verifying  : docker-scan-plugin-0.8.0-3.el7.x86_64                                              3/9 
      Verifying  : slirp4netns-0.4.3-4.el7_8.x86_64                                                   4/9 
      Verifying  : 2:container-selinux-2.119.2-1.911c772.el7_8.noarch                                 5/9 
      Verifying  : 3:docker-ce-20.10.7-3.el7.x86_64                                                   6/9 
      Verifying  : 1:docker-ce-cli-20.10.7-3.el7.x86_64                                               7/9 
      Verifying  : docker-ce-rootless-extras-20.10.7-3.el7.x86_64                                     8/9 
      Verifying  : fuse-overlayfs-0.7.2-6.el7_8.x86_64                                                9/9 
    
    Installed:
      docker-ce.x86_64 3:20.10.7-3.el7                                                                    
    
    Dependency Installed:
      container-selinux.noarch 2:2.119.2-1.911c772.el7_8 containerd.io.x86_64 0:1.4.6-3.1.el7            
      docker-ce-cli.x86_64 1:20.10.7-3.el7               docker-ce-rootless-extras.x86_64 0:20.10.7-3.el7
      docker-scan-plugin.x86_64 0:0.8.0-3.el7            fuse-overlayfs.x86_64 0:0.7.2-6.el7_8           
      fuse3-libs.x86_64 0:3.6.1-4.el7                    slirp4netns.x86_64 0:0.4.3-4.el7_8              
    
    Complete!
    [root@master yum.repos.d]# 
    [root@master ~]# systemctl enable docker --now
    Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
    [root@master ~]#
    
    [root@master ~]# docker info
    Client:
     Context:    default
     Debug Mode: false
     Plugins:
      app: Docker App (Docker Inc., v0.9.1-beta3)
      buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
      scan: Docker Scan (Docker Inc., v0.8.0)
    
    Server:
     Containers: 0
      Running: 0
      Paused: 0
      Stopped: 0
     Images: 4
     Server Version: 20.10.7
     Storage Driver: overlay2
      Backing Filesystem: xfs
      Supports d_type: true
      Native Overlay Diff: true
      userxattr: false
     Logging Driver: json-file
     Cgroup Driver: cgroupfs
     Cgroup Version: 1
     Plugins:
      Volume: local
      Network: bridge host ipvlan macvlan null overlay
      Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
     Swarm: inactive
     Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
     Default Runtime: runc
     Init Binary: docker-init
     containerd version: d71fcd7d8303cbf684402823e425e9dd2e99285d
     runc version: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
     init version: de40ad0
     Security Options:
      seccomp
       Profile: default
     Kernel Version: 3.10.0-957.el7.x86_64
     Operating System: CentOS Linux 7 (Core)
     OSType: linux
     Architecture: x86_64
     CPUs: 2
     Total Memory: 3.701GiB
     Name: master
     ID: 5FW3:5O7N:PZTJ:YUAT:GFXD:QEGA:GOA6:C2IE:I2FJ:FUQE:D2QT:QI6A
     Docker Root Dir: /var/lib/docker
     Debug Mode: false
     Registry: https://index.docker.io/v1/
     Labels:
     Experimental: false
     Insecure Registries:
      127.0.0.0/8
     Live Restore Enabled: false
    
    [root@master ~]#

## 配置加速器 ##

*  配置这个只是为了获取镜像更快一些罢了
 *  直接复制下面命令执行即可

cat > /etc/docker/daemon.json <<EOF { "registry-mirrors": ["https://frz7i079.mirror.aliyuncs.com"] } EOF

*  然后重启docker服务

[root@master ~]# systemctl restart docker
    [root@master ~]#

## 设置内核参数 ##

由于网桥工作于数据链路层，在iptables没有开启bridge-nf时，数据会直接经过网桥转发，结果就是对forward的设置失效。

cat <<EOF > /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 EOF

*  执行后如下

[root@master ~]# cat <<EOF > /etc/sysctl.d/k8s.conf 
    > net.bridge.bridge-nf-call-ip6tables = 1 
    > net.bridge.bridge-nf-call-iptables = 1 
    > net.ipv4.ip_forward = 1 
    > EOF
    [root@master ~]# 
    [root@master ~]# cat /etc/sysctl.d/k8s.conf
    net.bridge.bridge-nf-call-ip6tables = 1 
    net.bridge.bridge-nf-call-iptables = 1 
    net.ipv4.ip_forward = 1 
    [root@master ~]#

## 安装k8s ##

### 查看可用版本 ###

命令：`yum list --showduplicates kubeadm --disableexcludes=kubernetes`

[root@ccx k8s]# yum list --showduplicates kubeadm --disableexcludes=kubernetes
    Loaded plugins: fastestmirror, langpacks, product-id, search-disabled-repos, subscription-
                  : manager
    
    This system is not registered with an entitlement server. You can use subscription-manager
     to register.
    Loading mirror speeds from cached hostfile
    Available Packages
    kubeadm.x86_64                            1.6.0-0                               kubernetes
    kubeadm.x86_64                            1.6.1-0                               kubernetes
    kubeadm.x86_64                            1.6.2-0                               kubernetes
    kubeadm.x86_64                            1.6.3-0                               kubernetes
    kubeadm.x86_64                            1.6.4-0                               kubernetes
    kubeadm.x86_64                            1.6.5-0                               kubernetes
    kubeadm.x86_64                            1.6.6-0                               kubernetes
    kubeadm.x86_64                            1.6.7-0                               kubernetes
    kubeadm.x86_64                            1.6.8-0                               kubernetes
    kubeadm.x86_64                            1.6.9-0                               kubernetes
    kubeadm.x86_64                            1.6.10-0                              kubernetes
    kubeadm.x86_64                            1.6.11-0                              kubernetes
    kubeadm.x86_64                            1.6.12-0                              kubernetes
    kubeadm.x86_64                            1.6.13-0                              kubernetes
    kubeadm.x86_64                            1.7.0-0                               kubernetes
    kubeadm.x86_64                            1.7.1-0                               kubernetes
    kubeadm.x86_64                            1.7.2-0                               kubernetes
    kubeadm.x86_64                            1.7.3-1                               kubernetes
    kubeadm.x86_64                            1.7.4-0                               kubernetes
    kubeadm.x86_64                            1.7.5-0                               kubernetes
    kubeadm.x86_64                            1.7.6-1                               kubernetes
    kubeadm.x86_64                            1.7.7-1                               kubernetes
    kubeadm.x86_64                            1.7.8-1                               kubernetes
    kubeadm.x86_64                            1.7.9-0                               kubernetes
    kubeadm.x86_64                            1.7.10-0                              kubernetes
    kubeadm.x86_64                            1.7.11-0                              kubernetes
    kubeadm.x86_64                            1.7.14-0                              kubernetes
    kubeadm.x86_64                            1.7.15-0                              kubernetes
    kubeadm.x86_64                            1.7.16-0                              kubernetes
    kubeadm.x86_64                            1.8.0-0                               kubernetes
    kubeadm.x86_64                            1.8.0-1                               kubernetes
    kubeadm.x86_64                            1.8.1-0                               kubernetes
    kubeadm.x86_64                            1.8.2-0                               kubernetes
    kubeadm.x86_64                            1.8.3-0                               kubernetes
    kubeadm.x86_64                            1.8.4-0                               kubernetes
    kubeadm.x86_64                            1.8.5-0                               kubernetes
    kubeadm.x86_64                            1.8.6-0                               kubernetes
    kubeadm.x86_64                            1.8.7-0                               kubernetes
    kubeadm.x86_64                            1.8.8-0                               kubernetes
    kubeadm.x86_64                            1.8.9-0                               kubernetes
    kubeadm.x86_64                            1.8.10-0                              kubernetes
    kubeadm.x86_64                            1.8.11-0                              kubernetes
    kubeadm.x86_64                            1.8.12-0                              kubernetes
    kubeadm.x86_64                            1.8.13-0                              kubernetes
    kubeadm.x86_64                            1.8.14-0                              kubernetes
    kubeadm.x86_64                            1.8.15-0                              kubernetes
    kubeadm.x86_64                            1.9.0-0                               kubernetes
    kubeadm.x86_64                            1.9.1-0                               kubernetes
    kubeadm.x86_64                            1.9.2-0                               kubernetes
    kubeadm.x86_64                            1.9.3-0                               kubernetes
    kubeadm.x86_64                            1.9.4-0                               kubernetes
    kubeadm.x86_64                            1.9.5-0                               kubernetes
    kubeadm.x86_64                            1.9.6-0                               kubernetes
    kubeadm.x86_64                            1.9.7-0                               kubernetes
    kubeadm.x86_64                            1.9.8-0                               kubernetes
    kubeadm.x86_64                            1.9.9-0                               kubernetes
    kubeadm.x86_64                            1.9.10-0                              kubernetes
    kubeadm.x86_64                            1.9.11-0                              kubernetes
    kubeadm.x86_64                            1.10.0-0                              kubernetes
    kubeadm.x86_64                            1.10.1-0                              kubernetes
    kubeadm.x86_64                            1.10.2-0                              kubernetes
    kubeadm.x86_64                            1.10.3-0                              kubernetes
    kubeadm.x86_64                            1.10.4-0                              kubernetes
    kubeadm.x86_64                            1.10.5-0                              kubernetes
    kubeadm.x86_64                            1.10.6-0                              kubernetes
    kubeadm.x86_64                            1.10.7-0                              kubernetes
    kubeadm.x86_64                            1.10.8-0                              kubernetes
    kubeadm.x86_64                            1.10.9-0                              kubernetes
    kubeadm.x86_64                            1.10.10-0                             kubernetes
    kubeadm.x86_64                            1.10.11-0                             kubernetes
    kubeadm.x86_64                            1.10.12-0                             kubernetes
    kubeadm.x86_64                            1.10.13-0                             kubernetes
    kubeadm.x86_64                            1.11.0-0                              kubernetes
    kubeadm.x86_64                            1.11.1-0                              kubernetes
    kubeadm.x86_64                            1.11.2-0                              kubernetes
    kubeadm.x86_64                            1.11.3-0                              kubernetes
    kubeadm.x86_64                            1.11.4-0                              kubernetes
    kubeadm.x86_64                            1.11.5-0                              kubernetes
    kubeadm.x86_64                            1.11.6-0                              kubernetes
    kubeadm.x86_64                            1.11.7-0                              kubernetes
    kubeadm.x86_64                            1.11.8-0                              kubernetes
    kubeadm.x86_64                            1.11.9-0                              kubernetes
    kubeadm.x86_64                            1.11.10-0                             kubernetes
    kubeadm.x86_64                            1.12.0-0                              kubernetes
    kubeadm.x86_64                            1.12.1-0                              kubernetes
    kubeadm.x86_64                            1.12.2-0                              kubernetes
    kubeadm.x86_64                            1.12.3-0                              kubernetes
    kubeadm.x86_64                            1.12.4-0                              kubernetes
    kubeadm.x86_64                            1.12.5-0                              kubernetes
    kubeadm.x86_64                            1.12.6-0                              kubernetes
    kubeadm.x86_64                            1.12.7-0                              kubernetes
    kubeadm.x86_64                            1.12.8-0                              kubernetes
    kubeadm.x86_64                            1.12.9-0                              kubernetes
    kubeadm.x86_64                            1.12.10-0                             kubernetes
    kubeadm.x86_64                            1.13.0-0                              kubernetes
    kubeadm.x86_64                            1.13.1-0                              kubernetes
    kubeadm.x86_64                            1.13.2-0                              kubernetes
    kubeadm.x86_64                            1.13.3-0                              kubernetes
    kubeadm.x86_64                            1.13.4-0                              kubernetes
    kubeadm.x86_64                            1.13.5-0                              kubernetes
    kubeadm.x86_64                            1.13.6-0                              kubernetes
    kubeadm.x86_64                            1.13.7-0                              kubernetes
    kubeadm.x86_64                            1.13.8-0                              kubernetes
    kubeadm.x86_64                            1.13.9-0                              kubernetes
    kubeadm.x86_64                            1.13.10-0                             kubernetes
    kubeadm.x86_64                            1.13.11-0                             kubernetes
    kubeadm.x86_64                            1.13.12-0                             kubernetes
    kubeadm.x86_64                            1.14.0-0                              kubernetes
    kubeadm.x86_64                            1.14.1-0                              kubernetes
    kubeadm.x86_64                            1.14.2-0                              kubernetes
    kubeadm.x86_64                            1.14.3-0                              kubernetes
    kubeadm.x86_64                            1.14.4-0                              kubernetes
    kubeadm.x86_64                            1.14.5-0                              kubernetes
    kubeadm.x86_64                            1.14.6-0                              kubernetes
    kubeadm.x86_64                            1.14.7-0                              kubernetes
    kubeadm.x86_64                            1.14.8-0                              kubernetes
    kubeadm.x86_64                            1.14.9-0                              kubernetes
    kubeadm.x86_64                            1.14.10-0                             kubernetes
    kubeadm.x86_64                            1.15.0-0                              kubernetes
    kubeadm.x86_64                            1.15.1-0                              kubernetes
    kubeadm.x86_64                            1.15.2-0                              kubernetes
    kubeadm.x86_64                            1.15.3-0                              kubernetes
    kubeadm.x86_64                            1.15.4-0                              kubernetes
    kubeadm.x86_64                            1.15.5-0                              kubernetes
    kubeadm.x86_64                            1.15.6-0                              kubernetes
    kubeadm.x86_64                            1.15.7-0                              kubernetes
    kubeadm.x86_64                            1.15.8-0                              kubernetes
    kubeadm.x86_64                            1.15.9-0                              kubernetes
    kubeadm.x86_64                            1.15.10-0                             kubernetes
    kubeadm.x86_64                            1.15.11-0                             kubernetes
    kubeadm.x86_64                            1.15.12-0                             kubernetes
    kubeadm.x86_64                            1.16.0-0                              kubernetes
    kubeadm.x86_64                            1.16.1-0                              kubernetes
    kubeadm.x86_64                            1.16.2-0                              kubernetes
    kubeadm.x86_64                            1.16.3-0                              kubernetes
    kubeadm.x86_64                            1.16.4-0                              kubernetes
    kubeadm.x86_64                            1.16.5-0                              kubernetes
    kubeadm.x86_64                            1.16.6-0                              kubernetes
    kubeadm.x86_64                            1.16.7-0                              kubernetes
    kubeadm.x86_64                            1.16.8-0                              kubernetes
    kubeadm.x86_64                            1.16.9-0                              kubernetes
    kubeadm.x86_64                            1.16.10-0                             kubernetes
    kubeadm.x86_64                            1.16.11-0                             kubernetes
    kubeadm.x86_64                            1.16.11-1                             kubernetes
    kubeadm.x86_64                            1.16.12-0                             kubernetes
    kubeadm.x86_64                            1.16.13-0                             kubernetes
    kubeadm.x86_64                            1.16.14-0                             kubernetes
    kubeadm.x86_64                            1.16.15-0                             kubernetes
    kubeadm.x86_64                            1.17.0-0                              kubernetes
    kubeadm.x86_64                            1.17.1-0                              kubernetes
    kubeadm.x86_64                            1.17.2-0                              kubernetes
    kubeadm.x86_64                            1.17.3-0                              kubernetes
    kubeadm.x86_64                            1.17.4-0                              kubernetes
    kubeadm.x86_64                            1.17.5-0                              kubernetes
    kubeadm.x86_64                            1.17.6-0                              kubernetes
    kubeadm.x86_64                            1.17.7-0                              kubernetes
    kubeadm.x86_64                            1.17.7-1                              kubernetes
    kubeadm.x86_64                            1.17.8-0                              kubernetes
    kubeadm.x86_64                            1.17.9-0                              kubernetes
    kubeadm.x86_64                            1.17.11-0                             kubernetes
    kubeadm.x86_64                            1.17.12-0                             kubernetes
    kubeadm.x86_64                            1.17.13-0                             kubernetes
    kubeadm.x86_64                            1.17.14-0                             kubernetes
    kubeadm.x86_64                            1.17.15-0                             kubernetes
    kubeadm.x86_64                            1.17.16-0                             kubernetes
    kubeadm.x86_64                            1.17.17-0                             kubernetes
    kubeadm.x86_64                            1.18.0-0                              kubernetes
    kubeadm.x86_64                            1.18.1-0                              kubernetes
    kubeadm.x86_64                            1.18.2-0                              kubernetes
    kubeadm.x86_64                            1.18.3-0                              kubernetes
    kubeadm.x86_64                            1.18.4-0                              kubernetes
    kubeadm.x86_64                            1.18.4-1                              kubernetes
    kubeadm.x86_64                            1.18.5-0                              kubernetes
    kubeadm.x86_64                            1.18.6-0                              kubernetes
    kubeadm.x86_64                            1.18.8-0                              kubernetes
    kubeadm.x86_64                            1.18.9-0                              kubernetes
    kubeadm.x86_64                            1.18.10-0                             kubernetes
    kubeadm.x86_64                            1.18.12-0                             kubernetes
    kubeadm.x86_64                            1.18.13-0                             kubernetes
    kubeadm.x86_64                            1.18.14-0                             kubernetes
    kubeadm.x86_64                            1.18.15-0                             kubernetes
    kubeadm.x86_64                            1.18.16-0                             kubernetes
    kubeadm.x86_64                            1.18.17-0                             kubernetes
    kubeadm.x86_64                            1.18.18-0                             kubernetes
    kubeadm.x86_64                            1.18.19-0                             kubernetes
    kubeadm.x86_64                            1.18.20-0                             kubernetes
    kubeadm.x86_64                            1.19.0-0                              kubernetes
    kubeadm.x86_64                            1.19.1-0                              kubernetes
    kubeadm.x86_64                            1.19.2-0                              kubernetes
    kubeadm.x86_64                            1.19.3-0                              kubernetes
    kubeadm.x86_64                            1.19.4-0                              kubernetes
    kubeadm.x86_64                            1.19.5-0                              kubernetes
    kubeadm.x86_64                            1.19.6-0                              kubernetes
    kubeadm.x86_64                            1.19.7-0                              kubernetes
    kubeadm.x86_64                            1.19.8-0                              kubernetes
    kubeadm.x86_64                            1.19.9-0                              kubernetes
    kubeadm.x86_64                            1.19.10-0                             kubernetes
    kubeadm.x86_64                            1.19.11-0                             kubernetes
    kubeadm.x86_64                            1.19.12-0                             kubernetes
    kubeadm.x86_64                            1.20.0-0                              kubernetes
    kubeadm.x86_64                            1.20.1-0                              kubernetes
    kubeadm.x86_64                            1.20.2-0                              kubernetes
    kubeadm.x86_64                            1.20.4-0                              kubernetes
    kubeadm.x86_64                            1.20.5-0                              kubernetes
    kubeadm.x86_64                            1.20.6-0                              kubernetes
    kubeadm.x86_64                            1.20.7-0                              kubernetes
    kubeadm.x86_64                            1.20.8-0                              kubernetes
    kubeadm.x86_64                            1.21.0-0                              kubernetes
    kubeadm.x86_64                            1.21.1-0                              kubernetes
    kubeadm.x86_64                            1.21.2-0                              kubernetes
    [root@ccx k8s]#

### 安装 ###

*  这里安装的是1.21版本的

yum install -y kubelet-1.21.0-0 kubeadm-1.21.0-0 kubectl-1.21.0-0 --disableexcludes=kubernetes

*  我前面说过，我这3台机子是没有外网的，所以我是通过离线的方式安装的k8s包，解压命令如下了。

[root@master ~]# cd /k8s/
    [root@master k8s]# ls
    13b4e820d82ad7143d786b9927adc414d3e270d3d26d844e93eff639f7142e50-kubelet-1.21.0-0.x86_64.rpm
    conntrack-tools-1.4.4-7.el7.x86_64.rpm
    d625f039f4a82eca35f6a86169446afb886ed9e0dfb167b38b706b411c131084-kubectl-1.21.0-0.x86_64.rpm
    db7cb5cb0b3f6875f54d10f02e625573988e3e91fd4fc5eef0b1876bb18604ad-kubernetes-cni-0.8.7-0.x86_64.rpm
    dc4816b13248589b85ee9f950593256d08a3e6d4e419239faf7a83fe686f641c-kubeadm-1.21.0-0.x86_64.rpm
    libnetfilter_cthelper-1.0.0-11.el7.x86_64.rpm
    libnetfilter_cttimeout-1.0.0-7.el7.x86_64.rpm
    libnetfilter_queue-1.0.2-2.el7_2.x86_64.rpm
    socat-1.7.3.2-2.el7.x86_64.rpm
    [root@master k8s]# cp * node1:/k8s
    cp: target ‘node1:/k8s’ is not a directory
    [root@master k8s]# 
    [root@master k8s]# scp * node1:/k8s
    root@node1's password: 13b4e820d82ad7143d786b9927adc414d3e270d3d26d844e93eff639f7142e50-kubelet 100% 20MB 22.4MB/s 00:00 conntrack-tools-1.4.4-7.el7.x86_64.rpm 100% 187KB 21.3MB/s 00:00 d625f039f4a82eca35f6a86169446afb886ed9e0dfb167b38b706b411c131084-kubectl 100% 9774KB 22.3MB/s 00:00 db7cb5cb0b3f6875f54d10f02e625573988e3e91fd4fc5eef0b1876bb18604ad-kuberne 100% 19MB 21.9MB/s 00:00 dc4816b13248589b85ee9f950593256d08a3e6d4e419239faf7a83fe686f641c-kubeadm 100% 9285KB 21.9MB/s 00:00 libnetfilter_cthelper-1.0.0-11.el7.x86_64.rpm 100% 18KB 1.4MB/s 00:00 libnetfilter_cttimeout-1.0.0-7.el7.x86_64.rpm 100% 18KB 6.3MB/s 00:00 libnetfilter_queue-1.0.2-2.el7_2.x86_64.rpm 100% 23KB 8.6MB/s 00:00 socat-1.7.3.2-2.el7.x86_64.rpm 100% 290KB 23.1MB/s 00:00 [root@master k8s]# scp * node2:/k8s root@node2's password: 
    13b4e820d82ad7143d786b9927adc414d3e270d3d26d844e93eff639f7142e50-kubelet 100%   20MB  20.0MB/s   00:01    
    conntrack-tools-1.4.4-7.el7.x86_64.rpm                                   100%  187KB  19.9MB/s   00:00    
    d625f039f4a82eca35f6a86169446afb886ed9e0dfb167b38b706b411c131084-kubectl 100% 9774KB  21.2MB/s   00:00    
    db7cb5cb0b3f6875f54d10f02e625573988e3e91fd4fc5eef0b1876bb18604ad-kuberne 100%   19MB  21.4MB/s   00:00    
    dc4816b13248589b85ee9f950593256d08a3e6d4e419239faf7a83fe686f641c-kubeadm 100% 9285KB  19.6MB/s   00:00    
    libnetfilter_cthelper-1.0.0-11.el7.x86_64.rpm                            100%   18KB   5.9MB/s   00:00    
    libnetfilter_cttimeout-1.0.0-7.el7.x86_64.rpm                            100%   18KB   2.9MB/s   00:00    
    libnetfilter_queue-1.0.2-2.el7_2.x86_64.rpm                              100%   23KB   8.3MB/s   00:00    
    socat-1.7.3.2-2.el7.x86_64.rpm                                           100%  290KB  21.1MB/s   00:00    
    [root@master k8s]# rpm -ivhU * --nodeps --force
    Preparing...                          ################################# [100%]
    Updating / installing...
       1:socat-1.7.3.2-2.el7              ################################# [ 11%]
       2:libnetfilter_queue-1.0.2-2.el7_2 ################################# [ 22%]
       3:libnetfilter_cttimeout-1.0.0-7.el################################# [ 33%]
       4:libnetfilter_cthelper-1.0.0-11.el################################# [ 44%]
       5:conntrack-tools-1.4.4-7.el7      ################################# [ 56%]
       6:kubernetes-cni-0.8.7-0           ################################# [ 67%]
       7:kubelet-1.21.0-0                 ################################# [ 78%]
       8:kubectl-1.21.0-0                 ################################# [ 89%]
       9:kubeadm-1.21.0-0                 ################################# [100%]
    [root@master k8s]#

### 启动k8s服务 ###

命令：`systemctl enable kubelet --now`  
意思是加入开启启动并现在启动。

[root@master k8s]# systemctl enable kubelet --now
    Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
    [root@master k8s]# systemctl is-active kubelet
    activating
    [root@master k8s]#

# 安装kubeadmin【创建集群】【master上操作】 #

## 安装kubeadmin ##

### 命令说明 ###

*  命令：`kubeadm init --image-repository registry.aliyuncs.com/google_containers --kubernetes-version=v1.21.0 --pod-network-cidr=10.244.0.0/16` 【最后面的10.244是自定义的】
    
     *  `kubeadm init`：初始化集群
     *  `--image-repository registry.aliyuncs.com/google_containers`：需要镜像就从registry.aliyuncs.com/google\_containers这获取
     *  `--kubernetes-version=v1.21.0`指定版本
     *  `--pod-network-cidr=10.244.0.0/16`：指定pod的网段

### 命令执行后镜像一览 ###

*  安装前docker中是没有任何镜像的

[root@ccx ~]# docker images
    REPOSITORY   TAG       IMAGE ID   CREATED   SIZE
    [root@ccx ~]#

*  下载完毕以后docker会多出这些镜像  
    这些镜像补全，缺一个dns

[root@ccx ~]# docker images
    REPOSITORY                                                        TAG        IMAGE ID     
      CREATED         SIZEregistry.aliyuncs.com/google_containers/kube-apiserver            v1.21.0    4d217480042e 
      2 months ago    126MBregistry.aliyuncs.com/google_containers/kube-proxy                v1.21.0    38ddd85fe90e 
      2 months ago    122MBregistry.aliyuncs.com/google_containers/kube-controller-manager   v1.21.0    09708983cc37 
      2 months ago    120MBregistry.aliyuncs.com/google_containers/kube-scheduler            v1.21.0    62ad3129eca8 
      2 months ago    50.6MBregistry.aliyuncs.com/google_containers/pause                     3.4.1      0f8457a4c2ec 
      5 months ago    683kBregistry.aliyuncs.com/google_containers/etcd                      3.4.13-0   0369cf4303ff 
      10 months ago   253MB[root@ccx ~]# 
    [root@ccx ~]#

### 命令执行报错及处理 ###

*  执行命令以后最终报错如下

[root@ccx k8s]# kubeadm init --image-repository registry.aliyuncs.com/google_containers --
    kubernetes-version=v1.21.0 --pod-network-cidr=10.244.0.0/16[init] Using Kubernetes version: v1.21.0
    [preflight] Running pre-flight checks
    	[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. T
    he recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/[preflight] Pulling images required for setting up a Kubernetes cluster
    [preflight] This might take a minute or two, depending on the speed of your internet conne
    ction[preflight] You can also perform this action in beforehand using 'kubeadm config images pu ll'error execution phase preflight: [preflight] Some fatal errors occurred:
    	[ERROR ImagePull]: failed to pull image registry.aliyuncs.com/google_containers/co
    redns/coredns:v1.8.0: output: Error response from daemon: pull access denied for registry.aliyuncs.com/google_containers/coredns/coredns, repository does not exist or may require 'docker login': denied: requested access to the resource is denied, error: exit status 1
    [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore- preflight-errors=...`To see the stack trace of this error execute with --v=5 or higher
    [root@ccx k8s]#

*  原因是因为：`pull image registry.aliyuncs.com/google_containers/co redns/coredns:v1.8.0`下载这个报错了，其实这个镜像是在的，只是命名错了而已，所以通过其他方式下载这个/coredns包然后导入就行了。
 *  不想去找的可以下载上传的：`https://download.csdn.net/download/cuichongxin/19965486`
 *  导入命令：`docker load -i coredns-1.21.tar`  
    全部过程过程如下

[root@master kubeadm]# rz -E
    rz waiting to receive.
     zmodem trl+C ȡ
    
      100%   41594 KB 20797 KB/s 00:00:02       0 Errors
    
    [root@master kubeadm]# ls
    coredns-1.21.tar
    [root@master kubeadm]# docker load -i coredns-1.21.tar 
    225df95e717c: Loading layer  336.4kB/336.4kB
    69ae2fbf419f: Loading layer  42.24MB/42.24MB
    Loaded image: registry.aliyuncs.com/google_containers/coredns/coredns:v1.8.0
    [root@master kubeadm]# 
    [root@master kubeadm]# docker images | grep dns
    registry.aliyuncs.com/google_containers/coredns/coredns   v1.8.0    296a6d5035e2   8 months ago   42.5MB
    [root@master kubeadm]#

*  全部镜像如下

[root@master k8s]# docker images
    REPOSITORY                                                        TAG        IMAGE ID       CREATED         SIZE
    registry.aliyuncs.com/google_containers/kube-apiserver            v1.21.0    4d217480042e   2 months ago    126MB
    registry.aliyuncs.com/google_containers/kube-proxy                v1.21.0    38ddd85fe90e   2 months ago    122MB
    registry.aliyuncs.com/google_containers/kube-controller-manager   v1.21.0    09708983cc37   2 months ago    120MB
    registry.aliyuncs.com/google_containers/kube-scheduler            v1.21.0    62ad3129eca8   2 months ago    50.6MB
    registry.aliyuncs.com/google_containers/pause                     3.4.1      0f8457a4c2ec   5 months ago    683kB
    registry.aliyuncs.com/google_containers/coredns/coredns           v1.8.0     296a6d5035e2   8 months ago    42.5MB
    registry.aliyuncs.com/google_containers/etcd                      3.4.13-0   0369cf4303ff   10 months ago   253MB
    [root@master k8s]#

*  注意：`coredns-1.21.tar`这个包需要拷贝到3个节点上【master和node节点】

## 启动kubeadmin ##

当上面的镜像全部准备完毕以后，再次执行`kubeadm init --image-repository registry.aliyuncs.com/google_containers --kubernetes-version=v1.21.0 --pod-network-cidr=10.244.0.0/16`，最下面就会提示这些内容【现在没有启动成功】，说缺认证文件。

[addons] Applied essential addon: CoreDNS
    [addons] Applied essential addon: kube-proxy
    
    Your Kubernetes control-plane has initialized successfully!
    
    To start using your cluster, you need to run the following as a regular user:
    
      mkdir -p $HOME/.kube
      sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
      sudo chown $(id -u):$(id -g) $HOME/.kube/config
    
    Alternatively, if you are the root user, you can run:
    
      export KUBECONFIG=/etc/kubernetes/admin.conf
    
    You should now deploy a pod network to the cluster.
    Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
      https://kubernetes.io/docs/concepts/cluster-administration/addons/
    
    Then you can join any number of worker nodes by running the following on each as root:
    
    kubeadm join 192.168.59.142:6443 --token 29ivpt.l5x5zhqzca4n47kp \
            --discovery-token-ca-cert-hash sha256:5a9779259cea2b3a23fa6e713ed02f5b0c6eda2d75049b703dbec78587015e0b 
    [root@master k8s]#

*  现在我们可以根据提示复制执行下面命令即可：

[root@master k8s]# mkdir -p $HOME/.kube
    [root@master k8s]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    [root@master k8s]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
    [root@master k8s]#

*  上面3个命令执行完，就可以看到当前节点信息了  
    命令：`kubectl get nodes`

[root@master k8s]# kubectl get nodes
    NAME     STATUS     ROLES                  AGE     VERSION
    master   NotReady   control-plane,master   8m19s   v1.21.0
    [root@master k8s]#

## 查看当前集群信息 ##

*  命令：`kubeadm token create --print-join-command`  
    作用：在node节点上执行该命令，就可以加入该集群了

[root@master k8s]# 
    [root@master k8s]# kubeadm token create --print-join-command
    kubeadm join 192.168.59.142:6443 --token mguyry.aky5kho66cnjtlsl --discovery-token-ca-cert-hash sha256:5a9779259cea2b3a23fa6e713ed02f5b0c6eda2d75049b703dbec78587015e0b 
    [root@master k8s]#

# 加入集群【node节点上操作】 #

## 加入集群【kubeadmin】 ##

*  命令：就是master上执行`kubeadm token create --print-join-command`后的结果。  
    如，我在master上的结果是如下：  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2N1aWNob25neGlu_size_16_color_FFFFFF_t_70]
 *  那么我就在2个node节点都执行这结果即可：

[root@node1 k8s]# kubeadm join 192.168.59.142:6443 --token sqjhzj.7aloiqau86k8xq54 --discovery-token-ca-cert-hash sha256:5a9779259cea2b3a23fa6e713ed02f5b0c6eda2d75049b703dbec78587015e0b 
    [preflight] Running pre-flight checks
            [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
    [preflight] Reading configuration from the cluster...
    [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
    [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
    [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
    [kubelet-start] Starting the kubelet
    [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
    
    This node has joined the cluster:
    * Certificate signing request was sent to apiserver and a response was received.
    * The Kubelet was informed of the new secure connection details.
    
    Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
    
    [root@node1 k8s]#

*  执行完毕以后去master节点上执行`kubectl get nodes`是可以看到这2个节点信息的

[root@master ~]# kubectl get nodes
    NAME     STATUS     ROLES                  AGE     VERSION
    master   NotReady   control-plane,master   29m     v1.21.0
    node1    NotReady   <none>                 3m59s   v1.21.0
    node2    NotReady   <none>                 79s     v1.21.0
    [root@master ~]#

注：现在上面集群的STATUS状态是`NotReady`，是不正常的，因为确实网络插件，下面的calico安装完毕以后状态就正常了。

# 网络插件calico安装 #

## calico.yam文件获取 ##

*  网址：`https://docs.projectcalico.org/`  
    进入以后直接搜索`calico.yam`  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2N1aWNob25neGlu_size_16_color_FFFFFF_t_70 1]  
    下拉会有一个curl开头的下载网址，我们可以直接用复杂下载连接下载，也可以在linux上用wget下载，把curl换成wget，最后的-O去掉即可。  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2N1aWNob25neGlu_size_16_color_FFFFFF_t_70 2]

## 查看calico所需镜像并下载【3台主机同步】 ##

*  命令：`grep image calico.yaml`  
    我的环境是没外网的，rz是拷贝我下载镜像的命令。

[root@master ~]# rz -E
    rz waiting to receive.
     zmodem trl+C ȡ
    
      100%     185 KB  185 KB/s 00:00:01       0 Errors
    
    [root@master ~]# ls | grep calico.yaml 
    calico.yaml
    [root@master ~]# grep image calico.yaml
              image: docker.io/calico/cni:v3.19.1
              image: docker.io/calico/cni:v3.19.1
              image: docker.io/calico/pod2daemon-flexvol:v3.19.1
              image: docker.io/calico/node:v3.19.1
              image: docker.io/calico/kube-controllers:v3.19.1
    [root@master ~]#

*  然后去一台有网的主机上下载这些镜像  
    下载命令：`docker pull 上面5个image名`，如：`docker pull docker.io/calico/cni:v3.19.1`  
    5个镜像均需要下载完【下载很慢，可以在一台服务器上下载，然后打包镜像，拷贝到另外2个镜像上解压即可，不会的我博客中有一篇是docker镜像管理，可以去看看】
 *  如果你嫌下载慢【一共差不多400M】，可以使用我已经下载好的，链接：`https://download.csdn.net/download/cuichongxin/19988222`  
    使用方法如下

[root@master ~]# scp calico-3.19-img.tar node1:~
    root@node1's password: calico-3.19-img.tar 100% 381MB 20.1MB/s 00:18 [root@master ~]# [root@master ~]# scp calico-3.19-img.tar node2:~ root@node2's password: 
    calico-3.19-img.tar                                100%  381MB  21.1MB/s   00:18    
    [root@master ~]#
    [root@node1 ~]# docker load -i calico-3.19-img.tar 
    a4bf22d258d8: Loading layer  88.58kB/88.58kB
    d570c523a9c3: Loading layer  13.82kB/13.82kB
    b4d08d55eb6e: Loading layer  145.8MB/145.8MB
    Loaded image: calico/cni:v3.19.1
    01c2272d9083: Loading layer  13.82kB/13.82kB
    ad494b1a2a08: Loading layer   2.55MB/2.55MB
    d62fbb2a27c3: Loading layer  5.629MB/5.629MB
    2ac91a876a81: Loading layer  5.629MB/5.629MB
    c923b81cc4f1: Loading layer   2.55MB/2.55MB
    aad8b538640b: Loading layer  5.632kB/5.632kB
    b3e6b5038755: Loading layer  5.378MB/5.378MB
    Loaded image: calico/pod2daemon-flexvol:v3.19.1
    4d2bd71609f0: Loading layer  170.8MB/170.8MB
    dc8ea6c15e2e: Loading layer  13.82kB/13.82kB
    Loaded image: calico/node:v3.19.1
    ce740cb4fc7d: Loading layer  13.82kB/13.82kB
    fe82f23d6e35: Loading layer   2.56kB/2.56kB
    98738437610a: Loading layer  57.53MB/57.53MB
    0443f62e8f15: Loading layer  3.082MB/3.082MB
    Loaded image: calico/kube-controllers:v3.19.1
    [root@node1 ~]#

*  全部镜像准备完毕后如下：

[root@node1 ~]# docker images | grep ca
    calico/node                                                       v3.19.1    c4d75af7e098   6 weeks ago     168MB
    calico/pod2daemon-flexvol                                         v3.19.1    5660150975fb   6 weeks ago     21.7MB
    calico/cni                                                        v3.19.1    5749e8b276f9   6 weeks ago     146MB
    calico/kube-controllers                                           v3.19.1    5d3d5ddc8605   6 weeks ago     60.6MB
    registry.aliyuncs.com/google_containers/kube-scheduler            v1.21.0    62ad3129eca8   2 months ago    50.6MB
    [root@node1 ~]#

## 安装calico【master上操作】 ##

*  设置网段【master上操作】
    
     *  编辑配置文件 `calico.yaml`，将下面2行前面的\#和空格删掉【格式保持对齐】
     *  将value后面的192.168改成你初始化集群时候的网段，我用的是10.244。

[root@master ~]# cat calico.yaml | egrep -B 1 10.244
                - name: CALICO_IPV4POOL_CIDR
                  value: "10.244.0.0/16"
    [root@master ~]#

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2N1aWNob25neGlu_size_16_color_FFFFFF_t_70 3]

*  然后执行：`kubectl apply -f calico.yaml`

[root@master ~]# kubectl apply -f calico.yaml
    configmap/calico-config created
    customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
    customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
    customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
    customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
    customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
    customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
    customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
    customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
    customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
    customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
    customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
    customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
    customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
    customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
    customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
    clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
    clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
    clusterrole.rbac.authorization.k8s.io/calico-node created
    clusterrolebinding.rbac.authorization.k8s.io/calico-node created
    daemonset.apps/calico-node created
    serviceaccount/calico-node created
    deployment.apps/calico-kube-controllers created
    serviceaccount/calico-kube-controllers created
    Warning: policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
    poddisruptionbudget.policy/calico-kube-controllers created
    [root@master ~]#

*  现在执行 `kubectl get nodes`可以看到STATUS状态正常了

[root@master ~]# kubectl get nodes
    NAME     STATUS   ROLES                  AGE   VERSION
    master   Ready    control-plane,master   70m   v1.21.0
    node1    Ready    <none>                 45m   v1.21.0
    node2    Ready    <none>                 42m   v1.21.0
    [root@master ~]#

至此，集群的基本搭建就完成了。

查看calico装在那些节点上的，命令：`kubectl get pods -n kube-system -o wide` 【了解下就行】

[root@master ~]# kubectl get pods -n kube-system -o wide
    NAME                                       READY   STATUS    RESTARTS   AGE    IP               NODE     NOMINATED NODE   READINESS GATES
    calico-kube-controllers-78d6f96c7b-p4svs   1/1     Running   0          98m    10.244.219.67    master   <none>           <none>
    calico-node-cc4fc                          1/1     Running   18         65m    192.168.59.144   node2    <none>           <none>
    calico-node-stdfj                          1/1     Running   20         98m    192.168.59.142   master   <none>           <none>
    calico-node-zhhz7                          1/1     Running   1          98m    192.168.59.143   node1    <none>           <none>
    coredns-545d6fc579-6kb9x                   1/1     Running   0          167m   10.244.219.65    master   <none>           <none>
    coredns-545d6fc579-v74hg                   1/1     Running   0          167m   10.244.219.66    master   <none>           <none>
    etcd-master                                1/1     Running   1          167m   192.168.59.142   master   <none>           <none>
    kube-apiserver-master                      1/1     Running   1          167m   192.168.59.142   master   <none>           <none>
    kube-controller-manager-master             1/1     Running   11         167m   192.168.59.142   master   <none>           <none>
    kube-proxy-45qgd                           1/1     Running   1          65m    192.168.59.144   node2    <none>           <none>
    kube-proxy-fdhpw                           1/1     Running   1          167m   192.168.59.142   master   <none>           <none>
    kube-proxy-zf6nt                           1/1     Running   1          142m   192.168.59.143   node1    <none>           <none>
    kube-scheduler-master                      1/1     Running   12         167m   192.168.59.142   master   <none>           <none>
    [root@master ~]#

# kubectl子命令tab设置 #

*  状况： 现在除了kubectl可以tab，剩下的都是不能tab的，你可以先测试一下。

[root@master ~]# kubectl ge

*  查看kubectl的cmd

[root@master ~]# kubectl --help | grep bash
      completion    Output shell completion code for the specified shell (bash or zsh)
    [root@master ~]#

*  解决方法  
    下面应该看得懂吧，在配置文件中加入`source <(kubectl completion bash)`，source让其生效。然后就可以正常tab了

[root@master ~]# head -n 3 /etc/profile
    # /etc/profile
    
    source <(kubectl completion bash)
    [root@master ~]# 
    [root@master ~]# source /etc/profile
    [root@master ~]# 
    [root@master ~]# 
    [root@master ~]# kubectl get

*  注：要确保本地有completion这个包，如果没有，就安装，否则上面的操作完成是没有意义的。

[root@master ~]# rpm -qa | grep bash
    bash-completion-2.1-6.el7.noarch
    bash-4.2.46-31.el7.x86_64
    [root@master ~]#

# 删除集群节点【踢出集群】【master上操作】 #

*  需要两步操作
    
     *  1、踢出命令：`kubectl drain nodes_name --ignore-daemonsets` 【nodes\_name查看，执行：kubectl get nodes】  
        这个命令执行后，执行kubectl get nodes，STATUS状态会多一个SchedulingDisabled，这时候才可以删除。
     *  2、删除命令：`kubectl delete nodes nodes_name`
 *  如：我现在将node2踢出集群

[root@master ~]# kubectl drain node2 --ignore-daemonsets 
    node/node2 cordoned
    WARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-mw4zf, kube-system/kube-proxy-fqwtc
    node/node2 drained
    [root@master ~]# 
    [root@master ~]# kubectl get nodes
    NAME     STATUS                     ROLES                  AGE   VERSION
    master   Ready                      control-plane,master   87m   v1.21.0
    node1    Ready                      <none>                 62m   v1.21.0
    node2    Ready,SchedulingDisabled   <none>                 59m   v1.21.0
    [root@master ~]# 
    [root@master ~]# kubectl delete nodes node2
    node "node2" deleted
    [root@master ~]# 
    [root@master ~]# kubectl get nodes
    NAME     STATUS   ROLES                  AGE   VERSION
    master   Ready    control-plane,master   88m   v1.21.0
    node1    Ready    <none>                 63m   v1.21.0
    [root@master ~]#

*  删了后node节点重新加入集群会报错，处理方法见下面“node节点重新加入集群操作【node节点操作】”

# 集群初始化【master上操作】 #

*  流程
    
     *  1、先踢出所有集群【含自己master节点】
     *  2、清空集群所有配置  
        命令：`kubeadm reset` 回车输入`y`即可。【执行后集群的所有设置都没了】
 *  上面2个步骤执行完毕以后，从上面“安装kubeadmin【创建集群】【master上操作】”开始重新创建集群和加入集群操作即可【calico也需要重装】。  
    注：加入集群时候需要从master上重新查看集群信息，之前的不能用了。
 *  删了后node节点重新加入集群会报错，处理方法见下面“node节点重新加入集群操作【node节点操作】”

# node节点重新加入集群操作【node节点操作】 #

## 加入集群报错原因及内容 ##

如果我们的node节点被master踢出集群以后，直接重新加入集群的话，会报如下错误：

[root@node2 ~]# kubeadm join 192.168.59.142:6443 --token sqjhzj.7aloiqau86k8xq54 --discovery-token-ca-cert-hash sha256:5a9779259cea2b3a23fa6e713ed02f5b0c6eda2d75049b703dbec78587015e0b 
    [preflight] Running pre-flight checks
            [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
    error execution phase preflight: [preflight] Some fatal errors occurred:
            [ERROR FileAvailable--etc-kubernetes-kubelet.conf]: /etc/kubernetes/kubelet.conf already exists
            [ERROR Port-10250]: Port 10250 is in use
            [ERROR FileAvailable--etc-kubernetes-pki-ca.crt]: /etc/kubernetes/pki/ca.crt already exists
    [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
    To see the stack trace of this error execute with --v=5 or higher
    [root@node2 ~]#

## 处理方法 ##

node节点要重新加入集群，需要重置集群状态，命令：`kubeadm reset`，回车后输入y即可

[root@node2 ~]# kubeadm reset
    [reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
    [reset] Are you sure you want to proceed? [y/N]: y
    [preflight] Running pre-flight checks
    W0702 11:17:49.527296   60215 removeetcdmember.go:79] [reset] No kubeadm config, using etcd pod spec to get data directory
    [reset] No etcd config found. Assuming external etcd
    [reset] Please, manually reset etcd to prevent further issues
    [reset] Stopping the kubelet service
    [reset] Unmounting mounted directories in "/var/lib/kubelet"
    [reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
    [reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
    [reset] Deleting contents of stateful directories: [/var/lib/kubelet /var/lib/dockershim /var/run/kubernetes /var/lib/cni]
    
    The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d
    
    The reset process does not reset or clean up iptables rules or IPVS tables.
    If you wish to reset iptables, you must do so manually by using the "iptables" command.
    
    If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
    to reset your system's IPVS tables.
    
    The reset process does not clean your kubeconfig files and you must remove them manually.
    Please, check the contents of the $HOME/.kube/config file.
    [root@node2 ~]#

*  现在重新加入集群就没问题了

[root@node2 ~]# kubeadm join 192.168.59.142:6443 --token sqjhzj.7aloiqau86k8xq54 --discovery-token-ca-cert-hash sha256:5a9779259cea2b3a23fa6e713ed02f5b0c6eda2d75049b703dbec78587015e0b 
    [preflight] Running pre-flight checks
            [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
    [preflight] Reading configuration from the cluster...
    [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
    [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
    [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
    [kubelet-start] Starting the kubelet
    [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
    
    This node has joined the cluster:
    * Certificate signing request was sent to apiserver and a response was received.
    * The Kubelet was informed of the new secure connection details.
    
    Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
    
    [root@node2 ~]#

[yum_linux_rpm]: https://blog.csdn.net/cuichongxin/article/details/117962719?spm=1001.2014.3001.5501
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2N1aWNob25neGlu_size_16_color_FFFFFF_t_70]: /images/20221005/d3db1f97f4ed47fcadb131097ef4ce2a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2N1aWNob25neGlu_size_16_color_FFFFFF_t_70 1]: /images/20221005/beb8d7be1a5f472f811236977b5c6717.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2N1aWNob25neGlu_size_16_color_FFFFFF_t_70 2]: /images/20221005/88a74ab386d744fe8ca334ca4a716b7d.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2N1aWNob25neGlu_size_16_color_FFFFFF_t_70 3]: /images/20221005/10c993c778814ecea51cb9db86e96a54.png