从零开始搭建K8S--搭建K8S Ingress

骑猪看日落 2023-03-01 09:57 82阅读 0赞

Ingress是个什么鬼，网上资料很多（推荐[官方][Link 1]），大家自行研究。简单来讲，就是一个负载均衡的玩意，其主要用来解决使用NodePort暴露Service的端口时Node IP会漂移的问题。同时，若大量使用NodePort暴露主机端口，管理会非常混乱。

好的解决方案就是让外界通过域名去访问Service，而无需关心其Node IP及Port。那为什么不直接使用Nginx？这是因为在K8S集群中，如果每加入一个服务，我们都在Nginx中添加一个配置，其实是一个重复性的体力活，只要是重复性的体力活，我们都应该通过技术将它干掉。

Ingress就可以解决上面的问题，其包含两个组件Ingress Controller和Ingress：

*  Ingress  
    将Nginx的配置抽象成一个Ingress对象，每添加一个新的服务只需写一个新的Ingress的yaml文件即可
 *  Ingress Controller  
    将新加入的Ingress转化成Nginx的配置文件并使之生效

好了，废话不多，走你~

# 准备操作 #

## 官方文档 ##

人生苦短，不造轮子，本文将以官方的标准脚本为基础进行搭建，参考请戳[官方文档][Link 2]。官方文档中要求依次执行如下命令：

curl https:/ /raw.githubusercontent.com/kubernetes /ingress-nginx/master /deploy/namespace.yaml \ 
       | kubectl apply -f - 
       
       curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/default-backend.yaml \ 
       | kubectl apply -f - 
       
       curl https:/ /raw.githubusercontent.com/kubernetes /ingress-nginx/master /deploy/configmap.yaml \ 
       | kubectl apply -f - 
       
       curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/tcp-services-configmap.yaml \ 
       | kubectl apply -f - 
       
       curl https:/ /raw.githubusercontent.com/kubernetes /ingress-nginx/master /deploy/udp-services-configmap.yaml \ 
       | kubectl apply -f -

以上yaml文件创建Ingress用到的Namespace、ConfigMap，以及默认的后端`default-backend`。最关键的一点是，由于之前我们基于Kubeadm创建了K8S集群，则还必须执行：

curl https:/ /raw.githubusercontent.com/kubernetes /ingress-nginx/master /deploy/rbac.yaml \ 
       | kubectl apply -f - 
       
       curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/with-rbac.yaml \ 
       | kubectl apply -f -

这是由于Kubeadm创建的集群默认开启了RABC，因此Ingress也必须创建相应的RABC权限控制。

## 导入镜像 ##

但是，直接按照上述方式执行，我们的Ingress很可能会无法使用。所以，我们需要将上述Yaml文件全部`wget`下来，经过一些修改后才能执行`kubectl apply -f`创建。另外需要注意的是，这些yaml文件中提到的一些镜像，国内目前无法下载，如：

gcr.io/google_containers/defaultbackend:1.4 
       quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0

本人已经提前下载好，大家请戳：

地址：https: //pan.baidu.com/s/1N-bK9hI7JTZZB6AzmaT8PA 
       密码： 1a8a

拿到镜像后，在每个节点上执行如下命令导入镜像：

docker load < quay.io #kubernetes-ingress-controller#nginx-ingress-controller_0.14.0.tar 
       docker tag 452a96d81c30 quay.io/kubernetes-ingress-controller/nginx-ingress-controller: 0.14 .0 
       docker load < gcr.io #google_containers#defaultbackend.tar 
       docker tag 452a96d81c30 gcr.io/google_containers/defaultbackend

如上所示，导入镜像后，别忘记给打tag，否则镜像名称为<none>：

![Image 1][]

image.png

# 主要文件介绍 #

这里，我们先对一些重要的文件进行简单介绍。

## default-backend.yaml ##

`default-backend`的作用是，如果外界访问的域名不存在的话，则默认转发到`default-http-backend`这个Service，其会直接返回404：

apiVersion: extensions/v1beta1 
       kind: Deployment 
       metadata: 
       name: default-http-backend 
       labels: 
       app: default-http-backend 
       namespace: ingress-nginx 
       spec: 
       replicas: 1 
       selector: 
       matchLabels: 
       app: default-http-backend 
       template: 
       metadata: 
       labels: 
       app: default-http-backend 
       spec: 
       terminationGracePeriodSeconds: 60 
       containers: 
       - name: default-http-backend 
       # Any image is permissible as long as: 
       # 1. It serves a 404 page at / 
       # 2. It serves 200 on a /healthz endpoint 
       image: gcr.io/google_containers/defaultbackend: 1.4 
       livenessProbe: 
       httpGet: 
       path: /healthz 
       port: 8080 
       scheme: HTTP 
       initialDelaySeconds: 30 
       timeoutSeconds: 5 
       ports: 
       - containerPort: 8080 
       resources: 
       limits: 
       cpu: 10m 
       memory: 20Mi 
       requests: 
       cpu: 10m 
       memory: 20Mi 
       --- 
       
       apiVersion: v1 
       kind: Service 
       metadata: 
       name: default-http-backend 
       namespace: ingress-nginx 
       labels: 
       app: default-http-backend 
       spec: 
       ports: 
       - port: 80 
       targetPort: 8080 
       selector: 
       app: default-http-backend

## rbac.yaml ##

`rbac.yaml`负责Ingress的RBAC授权的控制，其创建了Ingress用到的ServiceAccount、ClusterRole、Role、RoleBinding、ClusterRoleBinding。在上文[《从零开始搭建Kubernetes集群》][Kubernetes]中，我们已对这些概念进行了简单介绍。

apiVersion: v1 
       kind: ServiceAccount 
       metadata: 
       name: nginx-ingress-serviceaccount 
       namespace: ingress-nginx 
       
       --- 
       
       apiVersion: rbac.authorization.k8s.io/v1beta1 
       kind: ClusterRole 
       metadata: 
       name: nginx-ingress-clusterrole 
       rules: 
       - apiGroups: 
       - "" 
       resources: 
       - configmaps 
       - endpoints 
       - nodes 
       - pods 
       - secrets 
       verbs: 
       - list 
       - watch 
       - apiGroups: 
       - "" 
       resources: 
       - nodes 
       verbs: 
       - get 
       - apiGroups: 
       - "" 
       resources: 
       - services 
       verbs: 
       - get 
       - list 
       - watch 
       - apiGroups: 
       - "extensions" 
       resources: 
       - ingresses 
       verbs: 
       - get 
       - list 
       - watch 
       - apiGroups: 
       - "" 
       resources: 
       - events 
       verbs: 
       - create 
       - patch 
       - apiGroups: 
       - "extensions" 
       resources: 
       - ingresses/ status 
       verbs: 
       - update 
       
       --- 
       
       apiVersion: rbac.authorization.k8s.io/v1beta1 
       kind: Role 
       metadata: 
       name: nginx-ingress- role 
       namespace: ingress-nginx 
       rules: 
       - apiGroups: 
       - "" 
       resources: 
       - configmaps 
       - pods 
       - secrets 
       - namespaces 
       verbs: 
       - get 
       - apiGroups: 
       - "" 
       resources: 
       - configmaps 
       resourceNames: 
       # Defaults to "<election-id>-<ingress-class>" 
       # Here: "<ingress-controller-leader>-<nginx>" 
       # This has to be adapted if you change either parameter 
       # when launching the nginx-ingress-controller. 
       - "ingress-controller-leader-nginx" 
       verbs: 
       - get 
       - update 
       - apiGroups: 
       - "" 
       resources: 
       - configmaps 
       verbs: 
       - create 
       - apiGroups: 
       - "" 
       resources: 
       - endpoints 
       verbs: 
       - get 
       
       --- 
       
       apiVersion: rbac.authorization.k8s.io/v1beta1 
       kind: RoleBinding 
       metadata: 
       name: nginx-ingress- role-nisa-binding 
       namespace: ingress-nginx 
       roleRef: 
       apiGroup: rbac.authorization.k8s.io 
       kind: Role 
       name: nginx-ingress- role 
       subjects: 
       - kind: ServiceAccount 
       name: nginx-ingress-serviceaccount 
       namespace: ingress-nginx 
       
       --- 
       
       apiVersion: rbac.authorization.k8s.io/v1beta1 
       kind: ClusterRoleBinding 
       metadata: 
       name: nginx-ingress-clusterrole-nisa-binding 
       roleRef: 
       apiGroup: rbac.authorization.k8s.io 
       kind: ClusterRole 
       name: nginx-ingress-clusterrole 
       subjects: 
       - kind: ServiceAccount 
       name: nginx-ingress-serviceaccount 
       namespace: ingress-nginx

## with-rbac.yaml ##

`with-rbac.yaml`是Ingress的核心，用于创建ingress-controller。前面提到过，ingress-controller的作用是将新加入的Ingress进行转化为Nginx的配置。

apiVersion: extensions/v1beta1 
       kind: Deployment 
       metadata: 
       name: nginx-ingress-controller 
       namespace: ingress-nginx 
       spec: 
       replicas: 1 
       selector: 
       matchLabels: 
       app: ingress-nginx 
       template: 
       metadata: 
       labels: 
       app: ingress-nginx 
       annotations: 
       prometheus.io/port: '10254' 
       prometheus.io/scrape: 'true' 
       spec: 
       serviceAccountName: nginx-ingress-serviceaccount 
       containers: 
       - name: nginx-ingress-controller 
       image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller: 0.14. 0 
       args: 
       - /nginx-ingress-controller 
       - -- default-backend-service=$(POD_NAMESPACE)/ default-http-backend 
       - --configmap=$(POD_NAMESPACE)/nginx-configuration 
       - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services 
       - --udp-services-configmap=$(POD_NAMESPACE)/udp-services 
       - --annotations-prefix=nginx.ingress.kubernetes.io 
       env: 
       - name: POD_NAME 
       valueFrom: 
       fieldRef: 
       fieldPath: metadata. name 
       - name: POD_NAMESPACE 
       valueFrom: 
       fieldRef: 
       fieldPath: metadata.namespace 
       ports: 
       - name: http 
       containerPort: 80 
       - name: https 
       containerPort: 443 
       livenessProbe: 
       failureThreshold: 3 
       httpGet: 
       path: /healthz 
       port: 10254 
       scheme: HTTP 
       initialDelaySeconds: 10 
       periodSeconds: 10 
       successThreshold: 1 
       timeoutSeconds: 1 
       readinessProbe: 
       failureThreshold: 3 
       httpGet: 
       path: /healthz 
       port: 10254 
       scheme: HTTP 
       periodSeconds: 10 
       successThreshold: 1 
       timeoutSeconds: 1 
       securityContext: 
       runAsNonRoot: false

如上，可以看到nginx-ingress-controller启动时传入了参数，分别为前面创建的default-backend-service以及configmap。

# 创建Ingress #

## 1.创建Ingress-controller ##

需要注意的是，官方提供的`with-rbac.yaml`文件不能直接使用，我们必须修改两处：

### 加入hostNetwork配置 ###

如下，在`serviceAccountName`上方添加`hostNetwork: true`:

spec: 
       hostNetwork: true 
       serviceAccountName: nginx-ingress-serviceaccount 
       containers: 
       - name: nginx-ingress-controller 
       image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0 
       args: 
       - /nginx-ingress-controller 
       - --default-backend-service=$(POD_NAMESPACE)/default-http-backend 
       - --configmap=$(POD_NAMESPACE)/nginx-configuration 
       - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services 
       - --udp-services-configmap=$(POD_NAMESPACE)/udp-services 
       - --annotations-prefix=nginx.ingress.kubernetes.io

配置`hostNetwork: true`是一种直接定义Pod网络的方式。定义后，Ingress-controller的IP就与宿主机k8s-node1一样(192.168.56.101)，并且端口80也是宿主机上的端口。这样，我们通过该`192.168.56.101:80`，就可以直接访问到Ingress-controller（实际上就是nginx），然后Ingress-controller则会转发我们的请求到相应后端。

### 加入环境变量 ###

在其env部分加入如下环境变量：

env: 
       - name: POD_NAME 
       valueFrom: 
       fieldRef: 
       fieldPath: metadata. name 
       - name: POD_NAMESPACE 
       valueFrom: 
       fieldRef: 
       fieldPath: metadata.namespace 
       - name: KUBERNETES_MASTER 
       value: http: //192.168.56.101:8080

否则，创建后会提示如下错误：

[root@k8s-node1 ingress]# kubectl describe pod nginx-ingress-controller -9fbd7596d-rt9sf -n ingress-nginx 
       省略前面... 
       Events: 
       Type Reason Age From Message 
       ---- ------ ---- ---- ------- 
       Normal Scheduled 30s default-scheduler Successfully assigned nginx-ingress-controller -9fbd7596d-rt9sf to k8s-node1 
       Normal SuccessfulMountVolume 30s kubelet, k8s-node1 MountVolume.SetUp succeeded for volume "nginx-ingress-serviceaccount-token-lq2dt" 
       Warning BackOff 21s kubelet, k8s-node1 Back-off restarting failed container 
       Normal Pulled 11s (x3 over 29s) kubelet, k8s-node1 Container image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0" already present on machine 
       Normal Created 11s (x3 over 29s) kubelet, k8s-node1 Created container 
       Warning Failed 10s (x3 over 28s) kubelet, k8s-node1 Error: failed to start container "nginx-ingress-controller": Error response from daemon: OCI runtime create failed: container_linux. go: 348: starting container process caused "exec: \"/nginx-ingress-controller\": stat /nginx-ingress-controller: no such file or directory": unknown

修改`with-rbac.yaml`后，使用`kubectl -f create`命令分别执行如下yaml文件，即可创建Ingress-controller：

![Image 1][]

创建成功后如下所示：

[root@k8s-node1 ingress]# kubectl get pod -n ingress-nginx -o wide 
       NAME READY STATUS RESTARTS AGE IP NODE 
       default-http-backend -5c6d95c48-pdjn9 1/ 1 Running 0 23s 192.168 .36 .81 k8s-node1 
       nginx-ingress-controller -547cd7d9cb-jmvpn 1/ 1 Running 0 8s 192.168 .36 .82 k8s-node1

## 2.创建自定义Ingress ##

有了ingress-controller，我们就可以创建自定义的Ingress了。这里已提前搭建好了Kibana服务，我们针对Kibana创建一个Ingress：

apiVersion: extensions/v1beta1 
       kind: Ingress 
       metadata: 
       name: kibana-ingress 
       namespace: default 
       
       spec: 
       rules: 
       - host: myk8s.com 
       http: 
       paths: 
       - path: / 
       backend: 
       serviceName: kibana 
       servicePort: 5601

其中：

*  rules中的host必须为域名，不能为IP，表示Ingress-controller的Pod所在主机域名，也就是Ingress-controller的IP对应的域名。
 *  paths中的path则表示映射的路径。如映射`/`表示若访问`myk8s.com`，则会将请求转发至Kibana的service，端口为5601。

创建成功后，查看：

[root@k8s-node1 ingress]# kubectl get ingress -o wide 
       NAME HOSTS ADDRESS PORTS AGE 
       kibana-ingress myk8s .com 80 6 s

我们再执行`kubectl exec nginx-ingress-controller-5b79cbb5c6-2zr7f -it cat /etc/nginx/nginx.conf -n ingress-nginx`，可以看到生成nginx配置，篇幅较长，各位自行筛选：

## start server myk8s.com 
       server { 
       server_name myk8s.com ; 
       
       listen 80; 
       
       listen [::]:80; 
       
       set $proxy_upstream_name "-"; 
       
       location /kibana { 
       
       log_by_lua_block { 
       
       } 
       
       port_in_redirect off; 
       
       set $proxy_upstream_name ""; 
       
       set $namespace "kube-system"; 
       set $ingress_name "dashboard-ingress"; 
       set $service_name "kibana"; 
       
       client_max_body_size "1m"; 
       
       proxy_set_header Host $best_http_host; 
       
       # Pass the extracted client certificate to the backend 
       
       # Allow websocket connections 
       proxy_set_header Upgrade $http_upgrade; 
       
       proxy_set_header Connection $connection_upgrade; 
       
       proxy_set_header X-Real-IP $the_real_ip; 
       
       proxy_set_header X-Forwarded-For $the_real_ip; 
       
       proxy_set_header X-Forwarded-Host $best_http_host; 
       proxy_set_header X-Forwarded-Port $pass_port; 
       proxy_set_header X-Forwarded-Proto $pass_access_scheme; 
       
       proxy_set_header X-Original-URI $request_uri; 
       
       proxy_set_header X-Scheme $pass_access_scheme; 
       
       # Pass the original X-Forwarded-For 
       proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for; 
       
       # mitigate HTTPoxy Vulnerability 
       # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/ 
       proxy_set_header Proxy ""; 
       
       # Custom headers to proxied server 
       
       proxy_connect_timeout 5s; 
       proxy_send_timeout 60s; 
       proxy_read_timeout 60s; 
       
       proxy_buffering "off"; 
       proxy_buffer_size "4k"; 
       proxy_buffers 4 "4k"; 
       proxy_request_buffering "on"; 
       
       proxy_http_version 1.1; 
       
       proxy_cookie_domain off; 
       proxy_cookie_path off; 
       
       # In case of errors try the next upstream server before returning an error 
       proxy_next_upstream error timeout invalid_header http_502 http_503 http_504; 
       proxy_next_upstream_tries 0; 
       
       # No endpoints available for the request 
       return 503; 
       
       } 
       
       location / { 
       
       log_by_lua_block { 
       
       } 
       
       port_in_redirect off; 
       
       set $proxy_upstream_name ""; 
       
       set $namespace "default"; 
       set $ingress_name "kibana-ingress"; 
       set $service_name "kibana"; 
       
       client_max_body_size "1m"; 
       
       proxy_set_header Host $best_http_host; 
       
       # Pass the extracted client certificate to the backend 
       
       # Allow websocket connections 
       proxy_set_header Upgrade $http_upgrade; 
       
       proxy_set_header Connection $connection_upgrade; 
       
       proxy_set_header X-Real-IP $the_real_ip; 
       
       proxy_set_header X-Forwarded-For $the_real_ip; 
       
       proxy_set_header X-Forwarded-Host $best_http_host; 
       proxy_set_header X-Forwarded-Port $pass_port; 
       proxy_set_header X-Forwarded-Proto $pass_access_scheme; 
       
       proxy_set_header X-Original-URI $request_uri; 
       
       proxy_set_header X-Scheme $pass_access_scheme; 
       
       # Pass the original X-Forwarded-For 
       proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for; 
       
       # mitigate HTTPoxy Vulnerability 
       # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/ 
       proxy_set_header Proxy ""; 
       
       # Custom headers to proxied server 
       
       proxy_connect_timeout 5s; 
       proxy_send_timeout 60s; 
       proxy_read_timeout 60s; 
       
       proxy_buffering "off"; 
       proxy_buffer_size "4k"; 
       proxy_buffers 4 "4k"; 
       proxy_request_buffering "on"; 
       
       proxy_http_version 1.1; 
       
       proxy_cookie_domain off; 
       proxy_cookie_path off; 
       
       # In case of errors try the next upstream server before returning an error 
       proxy_next_upstream error timeout invalid_header http_502 http_503 http_504; 
       proxy_next_upstream_tries 0; 
       
       # No endpoints available for the request 
       return 503; 
       
       } 
       
       } 
       ## end server myk8s.com

## 3.设置host ##

首先，我们需要在Ingress-controller的Pod所在主机上（这里为k8s-node1），将上面提到的域名`myk8s.com`追加入`/etc/hosts`文件：

192.168.56.101 myk8s.com

除此之外，如果想在自己的Windows物理机上使用浏览器访问kibana，也需要在`C:\Windows\System32\drivers\etc\hosts`文件内加入上述内容。设置后，分别在k8s-node1和物理机上测试无误即可：

![Image 1][]

# 测试 #

在Windows物理机上，使用Chrome访问`myk8s.com`，也就是相当于访问了`192.168.56.101:80`：

![Image 1][]

随意访问一个错误的地址`myk8s.com/abc`，返回预期的404：

![Image 1][]

[Link 1]: https://kubernetes.io/docs/concepts/services-networking/ingress/
[Link 2]: https://kubernetes.github.io/ingress-nginx/deploy/
[Image 1]: 
[Kubernetes]: https://blog.csdn.net/java_zyq/article/details/82178152