kubernetes(k8s)：Metrics-Server资源监控

逃离我推掉我的手 2023-02-13 03:22 29阅读 0赞

### 文章目录 ###

*  1. metrics server是什么
 *  2. metrics-server部署

# 1. metrics server是什么 #

Metrics-Server是集群核心监控数据的聚合器，用来替换之前的heapster。

容器相关的 **Metrics** 主要来自于 kubelet 内置的 **cAdvisor** 服务，有了Metrics-Server之后，用户就可以通过标准的 Kubernetes API 来访问到这些监控数据。

*  Metrics API 只可以查询当前的度量数据，并不保存历史数据。
 *  Metrics API URI 为 /apis/metrics.k8s.io/，在 k8s.io/metrics 维护。
 *  必须部署 metrics-server 才能使用该 API，metrics-server 通过调用 Kubelet Summary API 获取数据。

示例：

http://127.0.0.1:8001/apis/metrics.k8s.io/v1beta1/nodes
    http://127.0.0.1:8001/apis/metrics.k8s.io/v1beta1/nodes/<node-name>
    http://127.0.0.1:8001/apis/metrics.k8s.io/v1beta1/namespace/<namespace-name>/pods/<pod-name>

metrics server并不是 kube-apiserver的一部分，而是通过`aggregator` 这种插件机制，在独立部署的情况下同 kube-apiserver一起统一对外服务。

kube-aggregator 其实就是一个根据URL选择具体的API后端的代理服务器。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzNjk2OQ_size_16_color_FFFFFF_t_70]![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzNjk2OQ_size_16_color_FFFFFF_t_70 1]![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzNjk2OQ_size_16_color_FFFFFF_t_70 2]Metrics-server属于Core metrics(核心指标)，提供API metrics.k8s.io，仅提供Node和Pod的CPU和内存使用情况。而其他Custom Metrics(自定义指标)由`Prometheus等组件`来完成。

# 2. metrics-server部署 #

metrics-server官方网站：[https://github.com/kubernetes-sigs/metrics-server][https_github.com_kubernetes-sigs_metrics-server]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzNjk2OQ_size_16_color_FFFFFF_t_70 3]下载官方的配置文件

wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.6/components.yaml
    //修改镜像位置为私有仓库

![在这里插入图片描述][20200526231620272.png]

[kubeadm@server1 ~]$ cat components.yaml 
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: system:aggregated-metrics-reader
      labels:
        rbac.authorization.k8s.io/aggregate-to-view: "true"
        rbac.authorization.k8s.io/aggregate-to-edit: "true"
        rbac.authorization.k8s.io/aggregate-to-admin: "true"
    rules:
    - apiGroups: ["metrics.k8s.io"]
      resources: ["pods", "nodes"]
      verbs: ["get", "list", "watch"]
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: metrics-server:system:auth-delegator
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: system:auth-delegator
    subjects:
    - kind: ServiceAccount
      name: metrics-server
      namespace: kube-system
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: metrics-server-auth-reader
      namespace: kube-system
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: extension-apiserver-authentication-reader
    subjects:
    - kind: ServiceAccount
      name: metrics-server
      namespace: kube-system
    ---
    apiVersion: apiregistration.k8s.io/v1beta1
    kind: APIService
    metadata:
      name: v1beta1.metrics.k8s.io
    spec:
      service:
        name: metrics-server
        namespace: kube-system
      group: metrics.k8s.io
      version: v1beta1
      insecureSkipTLSVerify: true
      groupPriorityMinimum: 100
      versionPriority: 100
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: metrics-server
      namespace: kube-system
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: metrics-server
      namespace: kube-system
      labels:
        k8s-app: metrics-server
    spec:
      selector:
        matchLabels:
          k8s-app: metrics-server
      template:
        metadata:
          name: metrics-server
          labels:
            k8s-app: metrics-server
        spec:
          serviceAccountName: metrics-server
          volumes:
          # mount in tmp so we can safely use from-scratch images and/or read-only containers
          - name: tmp-dir
            emptyDir: { }
          containers:
          - name: metrics-server
            image: metrics-server-amd64:v0.3.6  // 我将metrics-server-amd64:v0.3.6镜像文件放到了我自己的私有仓库中，所以修改了镜像地址
            imagePullPolicy: IfNotPresent
            args:
              - --cert-dir=/tmp
              - --secure-port=4443
            ports:
            - name: main-port
              containerPort: 4443
              protocol: TCP
            securityContext:
              readOnlyRootFilesystem: true
              runAsNonRoot: true
              runAsUser: 1000
            volumeMounts:
            - name: tmp-dir
              mountPath: /tmp
          nodeSelector:
            kubernetes.io/os: linux
            kubernetes.io/arch: "amd64"
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: metrics-server
      namespace: kube-system
      labels:
        kubernetes.io/name: "Metrics-server"
        kubernetes.io/cluster-service: "true"
    spec:
      selector:
        k8s-app: metrics-server
      ports:
      - port: 443
        protocol: TCP
        targetPort: main-port
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: system:metrics-server
    rules:
    - apiGroups:
      - ""
      resources:
      - pods
      - nodes
      - nodes/stats
      - namespaces
      - configmaps
      verbs:
      - get
      - list
      - watch
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: system:metrics-server
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: system:metrics-server
    subjects:
    - kind: ServiceAccount
      name: metrics-server
      namespace: kube-system

[kubeadm@server1 ~]$ kubectl apply -f components.yaml  // 通过yaml文件自动部署
    clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
    clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
    rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
    apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
    serviceaccount/metrics-server created
    deployment.apps/metrics-server created
    service/metrics-server created
    clusterrole.rbac.authorization.k8s.io/system:metrics-server created
    clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created

![在这里插入图片描述][20200526231635312.png]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzNjk2OQ_size_16_color_FFFFFF_t_70 4]当前还不能使用，需要修改其它配置(**部署后还有不少坑需要填)**

查看pod日志:

**报错一：** `dial tcp: lookup server2 on 10.96.0.10:53: no such host`

[kubeadm@server1 ~]$ kubectl -n kube-system get pod -o wide // metrics-server运行成功，但是还无法使用
    NAME                              READY   STATUS    RESTARTS   AGE   IP             NODE      NOMINATED NODE   READINESS GATES
    metrics-server-7cdfcc6666-2xxd5   1/1     Running   0          62s   10.244.1.139   server3   <none>           <none>
    
    [kubeadm@server1 ~]$ kubectl -n kube-system logs metrics-server-7cdfcc6666-2xxd5  // 会出现无法解析的错误
    dial tcp: lookup server4 on 10.96.0.10:53: no such host
    //原因是没有内网的DNS服务器，所以metrics-server无法解析节点名字。可以直接修改coredns的configmap，讲各个节点的主机名加入到hosts中，这样所有Pod都可以从CoreDNS中解析各个节点的名字

![在这里插入图片描述][20200526232035923.png]这是因为没有内网的DNS服务器，所以metrics-server无法解析节点名字。可以直接修改coredns的configmap，将各个节点的主机名加入到hosts中，这样所有Pod都可以从CoreDNS中解析各个节点的名字。

**解决办法：在coredns中写入解析地址**

[kubeadm@server1 ~]$ kubectl -n kube-system edit cm coredns 
            ready
            hosts { 
               172.25.60.2 server2
               172.25.60.3 server3
               172.25.60.4 server4
               fallthrough
            }
            kubernetes cluster.local in-addr.arpa ip6.arpa {

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzNjk2OQ_size_16_color_FFFFFF_t_70 5]  
再查看日志出现错误2

[kubeadm@server2 ~]$ kubectl -n kube-system logs metrics-server-7cdfcc6666-2xxd5 
     // 再次查看日志，出现x509: certificate signed by unknown authority的错误

**报错2**： `x509: certificate signed by unknown authority`

解决方法：

*  Metric Server 支持一个参数 `--kubelet-insecure-tls`，可以跳过这一检查，但是官方不推荐使用。

**启用TLS Bootstrap 证书签发**

所有节点做以下操作:

[root@server1 ~]# vim /var/lib/kubelet/config.yaml  // 启用TLS Bootstrap 证书签发
    serverTLSBootstrap: true   // 在最后添加
    [root@server1 ~]# systemctl  restart kubelet

批准证书:

[kubeadm@server2 ~]$ kubectl get csr
    NAME        AGE    SIGNERNAME                      REQUESTOR             CONDITION
    csr-gjf9h   113s   kubernetes.io/kubelet-serving   system:node:server2   Pending
    
    [kubeadm@server2 ~]$ kubectl certificate approve csr-gjf9h  // 绑定证书(证书签名)
    certificatesigningrequest.certificates.k8s.io/csr-gjf9h approved
    
    [kubeadm@server2 ~]$ kubectl get csr
    NAME        AGE     SIGNERNAME                      REQUESTOR             CONDITION
    csr-gjf9h   5m14s   kubernetes.io/kubelet-serving   system:node:server2   Approved,Issued
    
    [kubeadm@server2 ~]$ kubectl top node  // server2上绑定了证书，但是server3和server4还没有
    NAME      CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%     
    server2   80m          4%     940Mi           49%         
    server3   <unknown>                           <unknown>               <unknown>               <unknown>               
    server4   <unknown>                           <unknown>               <unknown>               <unknown>

将server3和server4，启用TLS Bootstrap 证书签发

[root@server4 ~]# vim /var/lib/kubelet/config.yaml
    serverTLSBootstrap: true
    [root@server4 ~]# systemctl  restart kubelet
    
    [root@server3 ~]# vim /var/lib/kubelet/config.yaml 
    serverTLSBootstrap: true
    [root@server3 ~]# systemctl  restart kubelet

在server2上将server3和server4上的证书签名

[kubeadm@server2 ~]$ kubectl certificate approve csr-bgcws
    certificatesigningrequest.certificates.k8s.io/csr-bgcws approved
    
    [kubeadm@server2 ~]$ kubectl certificate approve csr-f8pkn
    certificatesigningrequest.certificates.k8s.io/csr-f8pkn approved
    
    [kubeadm@server2 ~]$ kubectl get csr
    NAME        AGE     SIGNERNAME                      REQUESTOR             CONDITION
    csr-bgcws   87s     kubernetes.io/kubelet-serving   system:node:server3   Approved,Issued
    csr-f8pkn   2m17s   kubernetes.io/kubelet-serving   system:node:server4   Approved,Issued
    csr-gjf9h   44m     kubernetes.io/kubelet-serving   system:node:server2   Approved,Issued
    
    [kubeadm@server2 ~]$ kubectl top node
    NAME      CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
    server2   93m          4%     946Mi           49%       
    server3   25m          2%     368Mi           19%       
    server4   40m          2%     466Mi           24%

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzNjk2OQ_size_16_color_FFFFFF_t_70 6]服务拉起  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzNjk2OQ_size_16_color_FFFFFF_t_70 7]有的版本会出现报错三，  
**报错3:** `Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)`

如果metrics-server正常启动，没有错误，应该就是网络问题。修改metrics-server的Pod 网络模式：  
`hostNetwork: true`

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzNjk2OQ_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200526224142387.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzNjk2OQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzNjk2OQ_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/2020052622422669.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzNjk2OQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzNjk2OQ_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/2020052622424575.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzNjk2OQ==,size_16,color_FFFFFF,t_70
[https_github.com_kubernetes-sigs_metrics-server]: https://github.com/kubernetes-sigs/metrics-server
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzNjk2OQ_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20200526231518596.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzNjk2OQ==,size_16,color_FFFFFF,t_70
[20200526231620272.png]: https://img-blog.csdnimg.cn/20200526231620272.png
[20200526231635312.png]: https://img-blog.csdnimg.cn/20200526231635312.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzNjk2OQ_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20200526231720560.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzNjk2OQ==,size_16,color_FFFFFF,t_70
[20200526232035923.png]: https://img-blog.csdnimg.cn/20200526232035923.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzNjk2OQ_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/2020052623222295.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzNjk2OQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzNjk2OQ_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/20200526232745335.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzNjk2OQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzNjk2OQ_size_16_color_FFFFFF_t_70 7]: https://img-blog.csdnimg.cn/20200526232802239.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzNjk2OQ==,size_16,color_FFFFFF,t_70