手把手教你使用K8S部署Apollo高可用

ゞ浴缸里的玫瑰 2023-06-23 05:51 87阅读 0赞

## 一、下载源码 ##

[Apollo-git地址][Apollo-git]

## 二、创建数据库 ##

按照官网wiki创建数据库

## 三、修改configservice配置 ##

### 1. 修改config/apollo-configservice.conf ###

`只修改下面内容，调整容器内日志目录`

> LOG\_FOLDER=~/opt/logs/config/

### 2. 修改config/application-github.properties 【可改可不改，建议修改防止意外】 ###

`增加如下配置，禁用apollo的eureka，本次部署使用独立eureka集群`

> apollo.eureka.server.enabled=false

### 3. 修改resources/application.yml ###

`修改端口号，避免端口冲突，修改日志目录为config/apollo-configservice.conf中配置目录，指定文件名`

server:
      port: 28080
    
    logging:
      file: ～/opt/logs/config/apollo-configservice.log

### 4. 修改bootstrap.yml ###

`按注释修改，其余不用动`

eureka:
      instance:
        # 修改此配置如下 30003端口号在第九步中说明 k8s-node-ip替换为自己k8s节点ip
        homePageUrl: http://k8s-node-ip:30003
        # 修改此配置如下
        preferIpAddress: false
    
    # 增加此属性， 禁用apollo自带的eureka
    apollo:
      eureka:
        server:
          enabled: false

### 5. 修改configservice.properties ###

`只修改这两个属性，与resources/application.yml对应`

> server.port= 28080  
> logging.file= ～/opt/logs/config/apollo-configservice.log

### 6. 修改scripts/startup.sh ###

`只修改这两个属性，与上面对应`

## Adjust log dir if necessary
    LOG_DIR=~/opt/logs/config
    ## Adjust server port if necessary
    SERVER_PORT=${SERVER_PORT:=28080}

## 四、修改adminservier配置 ##

### 1. 修改config/apollo-adminservice.conf ###

`日志路径修改为当前用户目录下opt/logs/admin/文件夹内`

> LOG\_FOLDER=~/opt/logs/portal/

### 2. 修改config/application-github.properties 【可改可不改，建议修改防止意外】 ###

`增加如下配置，禁用apollo的eureka，本次部署使用独立eureka集群`

> apollo.eureka.server.enabled=false

### 3. 修改resources/adminservice.properties ###

`修改端口号，避免端口冲突，修改日志目录为config/apollo-adminservice.conf中配置目录，指定文件名`

> server.port= 28090  
> logging.file= ~/opt/logs/admin/apollo-adminservice.log

### 4. 修改resources/application.yml ###

`修改端口号，避免端口冲突，修改日志目录为config/apollo-adminservice.conf中配置目录，指定文件名`

server:
      port: 28090
      
    logging:
      file: ~/opt/logs/admin/apollo-adminservice.log

### 5. 修改resources/bootstrap.yml ###

`按注释修改，其余不用动`

eureka:
      instance:
        # 修改此配置如下 28091端口号在第十步中说明 k8s-node-ip替换为自己k8s节点ip
        homePageUrl: http://k8s-node-ip:28091
        # 修改此配置如下
        preferIpAddress: false
        
    # 增加此属性， 禁用apollo自带的eureka
    apollo:
      eureka:
        server:
          enabled: false

### 6. 修改scripts/startup.sh ###

`只修改这两个属性，与上面对应`

## Adjust log dir if necessary
    LOG_DIR=~/opt/logs/admin
    ## Adjust server port if necessary
    SERVER_PORT=${SERVER_PORT:=28090}

## 五、修改portal配置 ##

### 1. 修改config/apollo-portal.conf ###

`日志路径修改为当前用户目录下opt/logs/portal/文件夹内`

> LOG\_FOLDER=~/opt/logs/portal/

### 2. 修改resources/apollo-env.properties【可改可不改，建议修改】 ###

`修改这里的端口号为configservice服务的端口号`

> local.meta=http://localhost:28080

### 3. 修改resources/application.yml ###

`修改端口号，避免端口冲突，修改日志目录为config/apollo-portalservice.conf中配置目录，指定文件名`

server:
      port: 28070
      
    logging:
      file: ~/opt/logs/portal/apollo-portal.log

### 4. 修改scripts/startup.sh ###

## Adjust log dir if necessary
    LOG_DIR=~/opt/logs/portal
    ## Adjust server port if necessary
    SERVER_PORT=${SERVER_PORT:=28070}

## 六、修改Dockerfile内容 ##

### 1. 修改scripts/apollo-on/kubernetes/apollo-config-server/Dockerfile ###

`只修改下面属性值，这里重新指定k8s中服务命名空间`

> ENV APOLLO\_CONFIG\_SERVICE\_NAME=“service-apollo-config-server.apollo”

### 2. 修改scripts/apollo-on/kubernetes/apollo-config-server/scripts/startup-kubernetes.sh ###

`只修改端口号与日志存放目录`

## Adjust log dir if necessary
    LOG_DIR=~/opt/logs/config
    ## Adjust server port if necessary
    SERVER_PORT=28080

### 3. 修改scripts/apollo-on/kubernetes/apollo-admin-server/Dockerfile ###

`只修改下面属性值，这里重新指定k8s中服务命名空间`

> ENV APOLLO\_ADMIN\_SERVICE\_NAME=“service-apollo-admin-server.apollo”

### 4. 修改scripts/apollo-on/kubernetes/apollo-admin-server/scripts/startup-kubernetes.sh ###

`只修改端口号与日志存放目录`

## Adjust log dir if necessary
    LOG_DIR=~/opt/logs/apollo-admin-server
    ## Adjust server port if necessary
    SERVER_PORT=28090

### 5. 修改scripts/apollo-on/kubernetes/apollo-portal-server/Dockerfile ###

`只修改下面属性值，这里重新指定k8s中服务命名空间`

> ENV APOLLO\_PORTAL\_SERVICE\_NAME=“service-apollo-portal-server.apollo”

### 6. 修改scripts/apollo-on/kubernetes/apollo-portal-server/scripts/startup-kubernetes.sh ###

`只修改端口号与日志存放目录`

## Adjust log dir if necessary
    LOG_DIR=~/opt/logs/apollo-portal-server
    ## Adjust server port if necessary
    SERVER_PORT=28070

## 七、执行编译脚本 ##

在scripts/目录下运行./build.sh

## 八、编译镜像并push到docker仓库 ##

### 1. copy并重命名jar包 ###

1.  解压apollo-configservice/target/apollo-configservice-1.6.0-SNAPSHOT-github.zip, 将apollo-configservice-1.6.0-SNAPSHOT.jar复制到scripts/apollo-on/kubernetes/apollo-config-server/，并重命名为apollo-configservice.jar
2.  解压apollo-adminservice/target/apollo-adminservice-1.6.0-SNAPSHOT-github.zip, 将apollo-adminservice-1.6.0-SNAPSHOT.jar复制到scripts/apollo-on/kubernetes/apollo-admin-server/，并重命名为apollo-adminservice.jar
3.  解压apollo-portalservice/target/apollo-portalservice-1.6.0-SNAPSHOT-github.zip, 将apollo-portalservice-1.6.0-SNAPSHOT.jar复制到scripts/apollo-on/kubernetes/apollo-portal-server/，并重命名为apollo-portalservice.jar

### 2. 编译镜像并push ###

在以上jar包对应目录下分别执行：

> docker build -t imageName:tag .  
> docker push imageName:tag  
> `注意，build命令后有个点`

## 九、apollo-config-server.yml ##

# 在pod创建之前先创建configMap
    # pod可以通过三种方式使用configMap
    # 1. 将configMap中的数据设置为环境变量
    # 2. 将configMap中的数据设置为命令行参数
    # 3. 使用volume将configMap作为文件或目录挂载
    kind: ConfigMap
    apiVersion: v1
    metadata:
      namespace: apollo                                                 
      name: configmap-apollo-config-server-dev
    data:        
      # 【注意这里数据库连接与eureka连接在不同环境都需要切换】目前由于资源问题只提供dev环境，
      # 数据库通过ip连接 【改成自己的数据库连接】
      # eureka通过config-server-pod-name-index.meta-server-service-name:prot/eureka/指定 【没通过k8s部署的直接用自己的eureka服务】 
      application-github.properties: | spring.datasource.url = jdbc:mysql://mysql-server-ip:3306/ApolloConfigDB_Dev?characterEncoding=utf8 spring.datasource.username = username spring.datasource.password = password eureka.service.url = http://eureka-service-0.eureka-service.default:9008/eureka/,http://eureka-service-1.eureka-service.default:9008/eureka/,http://eureka-service-2.eureka-service.default:9008/eureka/
    
    ---
    kind: Service
    apiVersion: v1
    metadata:
      namespace: apollo
      name: service-apollo-meta-server-dev
      labels:
        app: service-apollo-meta-server-dev
    spec:
      ports:
        - protocol: TCP
          port: 28080 # 设定Serivce对外提供服务的端口
          targetPort: 28080 # 设定容器(Pod)的端口,即Pod网络的端口
          nodePort: 30003 # 将meta-server服务对节点外开发，让开发机器也能访问到k8s中的apollo
      selector:
        app: pod-apollo-config-server-dev
      type: NodePort
    # clusterIP: None
      sessionAffinity: ClientIP
    
    ---
    kind: Service
    apiVersion: v1
    metadata:
      namespace: apollo
      name: service-apollo-config-server-dev
      labels:
        app: service-apollo-config-server-dev
    spec:
      ports:
        - protocol: TCP
          port: 28080
          targetPort: 28080
          nodePort: 30002 # 它仅在type为NodePort时才需要指定
      selector:
        app: pod-apollo-config-server-dev 
      type: NodePort                          # NodePort类型的service, 可以让k8s节点外的主机都能访问到
      sessionAffinity: ClientIP
    
    
    # RC、Deployment、DaemonSet都是面向无状态的服务，它们所管理的Pod的IP、名字，启停顺序等都是随机的
    # StatefulSet是有状态的集合，管理所有有状态的服务
    # StatefulSet本质上是Deployment的一种变体，在v1.9版本中已成为GA版本，它为了解决有状态服务的问题，它所管理的Pod拥有固定的Pod名称，启停顺序
    # 在StatefulSet中，Pod名字称为网络标识(hostname)，还必须要用到共享存储
    ---
    kind: StatefulSet
    apiVersion: apps/v1
    metadata:
      namespace: apollo
      name: statefulset-apollo-config-server-dev
      labels:
        app: statefulset-apollo-config-server-dev
    spec:
      serviceName: service-apollo-meta-server-dev               # 声明它属于哪个Headless Service.
      replicas: 3
      selector:
        matchLabels:
          app: pod-apollo-config-server-dev                     # has to match .spec.template.metadata.labels
      updateStrategy:
        type: RollingUpdate
      template:
        metadata:
          labels:
            app: pod-apollo-config-server-dev                   # has to match .spec.selector.matchLabels
        spec:
          affinity:
            podAntiAffinity:
              preferredDuringSchedulingIgnoredDuringExecution:
              - weight: 100
                podAffinityTerm:
                  labelSelector:
                    matchExpressions:
                    - key: app
                      operator: In
                      values:
                      - pod-apollo-config-server-dev
                  topologyKey: kubernetes.io/hostname
    
          volumes:
            - name: volume-configmap-apollo-config-server-dev
              configMap:
                name: configmap-apollo-config-server-dev
                items:
                  - key: application-github.properties
                    path: application-github.properties
          
          containers:
            - image: imageName:tag   # 这里需要改成自己打的镜像
              securityContext:
                privileged: true
              imagePullPolicy: IfNotPresent
              name: container-apollo-config-server-dev
              ports:
                - protocol: TCP
                  containerPort: 28080
              resources:
                limits:
                  memory: 512Mi  # 限制内存为512M
              volumeMounts:
                - name: volume-configmap-apollo-config-server-dev
                  mountPath: /apollo-config-server/config/application-github.properties
                  subPath: application-github.properties
              
              env:
                - name: APOLLO_CONFIG_SERVICE_NAME
                  value: "service-apollo-config-server-dev.apollo"
              
              readinessProbe:
                tcpSocket:
                  port: 28080
                initialDelaySeconds: 10
                periodSeconds: 5
              
              livenessProbe:
                tcpSocket:
                  port: 28080
                initialDelaySeconds:  120
                periodSeconds: 10
              
          dnsPolicy: ClusterFirst
          restartPolicy: Always

## 十、apollo-admin-server.yml ##

# 在pod创建之前先创建configMap
    # pod可以通过三种方式使用configMap
    # 1. 将configMap中的数据设置为环境变量
    # 2. 将configMap中的数据设置为命令行参数
    # 3. 使用volume将configMap作为文件或目录挂载
    kind: ConfigMap
    apiVersion: v1
    metadata:
      namespace: apollo                                    
      name: configmap-apollo-admin-server-dev
    data:
      # 【注意这里数据库连接与eureka连接在不同环境都需要切换】目前由于资源问题只提供dev环境，
      # 数据库通过ip连接 【改成自己的数据库连接】
      # eureka通过config-server-pod-name-index.meta-server-service-name:prot/eureka/指定 【没通过k8s部署的直接用自己的eureka服务】 
      application-github.properties: | spring.datasource.url = jdbc:mysql://mysql-server-ip:3306/ApolloConfigDB_Dev?characterEncoding=utf8 spring.datasource.username = username spring.datasource.password = password eureka.service.url = http://eureka-service-0.eureka-service.default:9008/eureka/,http://eureka-service-1.eureka-service.default:9008/eureka/,http://eureka-service-2.eureka-service.default:9008/eureka/
    
    ---
    kind: Service
    apiVersion: v1
    metadata:
      namespace: apollo
      name: service-apollo-admin-server-dev
      labels:
        app: service-apollo-admin-server-dev
    spec:
      ports:
        - protocol: TCP
          port: 28090 # 设定Serivce对外提供服务的端口
          targetPort: 28090 # 设定容器(Pod)的端口,即Pod网络的端口
          nodePort: 28091
      selector:
        app: pod-apollo-admin-server-dev
      type: NodePort
      # clusterIP: None
      sessionAffinity: ClientIP
    
    # RC、Deployment、DaemonSet都是面向无状态的服务，它们所管理的Pod的IP、名字，启停顺序等都是随机的
    # StatefulSet是有状态的集合，管理所有有状态的服务
    # StatefulSet本质上是Deployment的一种变体，在v1.9版本中已成为GA版本，它为了解决有状态服务的问题，它所管理的Pod拥有固定的Pod名称，启停顺序
    # 在StatefulSet中，Pod名字称为网络标识(hostname)，还必须要用到共享存储
    ---
    kind: Deployment
    apiVersion: apps/v1
    metadata:
      namespace: apollo
      name: deployment-apollo-admin-server-dev
      labels:
        app: deployment-apollo-admin-server-dev
    spec:
      replicas: 3
      selector:
        matchLabels:
          app: pod-apollo-admin-server-dev                     # has to match .spec.template.metadata.labels
      strategy:
        rollingUpdate:
          maxSurge: 1
          maxUnavailable: 1
        type: RollingUpdate
      template:
        metadata:
          labels:
            app: pod-apollo-admin-server-dev                   # has to match .spec.selector.matchLabels
        spec:
          affinity:
            podAntiAffinity:
              preferredDuringSchedulingIgnoredDuringExecution:
              - weight: 100
                podAffinityTerm:
                  labelSelector:
                    matchExpressions:
                    - key: app
                      operator: In
                      values:
                      - pod-apollo-admin-server-dev
                  topologyKey: kubernetes.io/hostname
    
          volumes:
            - name: volume-configmap-apollo-admin-server-dev
              configMap:
                name: configmap-apollo-admin-server-dev
                items:
                  - key: application-github.properties
                    path: application-github.properties
    
          containers:
            - image: imageName:tag
              securityContext:
                privileged: true
              imagePullPolicy: IfNotPresent
              name: container-apollo-admin-server-dev
              ports:
                - protocol: TCP
                  containerPort: 28090
              resources:
                limits:
                  memory: 512Mi
              volumeMounts:
                - name: volume-configmap-apollo-admin-server-dev
                  mountPath: /apollo-admin-server/config/application-github.properties
                  subPath: application-github.properties
              
              env:
                - name: APOLLO_CONFIG_SERVICE_NAME
                  value: "service-apollo-admin-server-dev.apollo"
              
              readinessProbe:
                tcpSocket:
                  port: 28090
                initialDelaySeconds: 10
                periodSeconds: 5
              
              livenessProbe:
                tcpSocket:
                  port: 28090
                initialDelaySeconds:  120
                periodSeconds: 10
              
          dnsPolicy: ClusterFirst
          restartPolicy: Always

## 十一、apollo-portal-server.yml ##

---
    # configmap for apollo-portal-server
    kind: ConfigMap
    apiVersion: v1
    metadata:
      namespace: apollo
      name: configmap-apollo-portal-server
    data:
      # 数据库通过ip连接 【改成自己的数据库连接】
      application-github.properties: | spring.datasource.url = jdbc:mysql://mysql-server-ip:3306/ApolloPortalDB?characterEncoding=utf8 spring.datasource.username = username spring.datasource.password = password
      apollo-env.properties: | dev.meta=http://statefulset-apollo-config-server-dev-0.service-apollo-meta-server-dev:28080,http://statefulset-apollo-config-server-dev-1.service-apollo-meta-server-dev:28080,http://statefulset-apollo-config-server-dev-2.service-apollo-meta-server-dev:28080
    # 资源限制，现在只使用dev环境
    # fat.meta=http://statefulset-apollo-config-server-fat-0.service-apollo-meta-server-fat:28080,http://statefulset-apollo-config-server-fat-1.service-apollo-meta-server-fat:28080,http://statefulset-apollo-config-server-fat-2.service-apollo-meta-server-fat:28080
    # uat.meta=http://service-apollo-config-server-test-beta.sre:8080
    # pro.meta=http://service-apollo-config-server-prod.sre:8080
    
    ---
    kind: Service
    apiVersion: v1
    metadata:
      namespace: apollo
      name: service-apollo-portal-server
      labels:
        app: service-apollo-portal-server
    spec:
      ports:
        - protocol: TCP
          port: 28070
          targetPort: 28070
          nodePort: 30001
      selector:
        app: pod-apollo-portal-server
      type: NodePort
      # portal session 保持
      sessionAffinity: ClientIP
    
    # 这里使用Deployment
    ---
    kind: Deployment
    apiVersion: apps/v1
    metadata:
      namespace: apollo
      name: deployment-apollo-portal-server
      labels:
        app: deployment-apollo-portal-server
    spec:
      # 3 个实例
      replicas: 3
      selector:
        matchLabels:
          app: pod-apollo-portal-server
      strategy:
        rollingUpdate:
          maxSurge: 1
          maxUnavailable: 1
        type: RollingUpdate
      template:
        metadata:
          labels:
            app: pod-apollo-portal-server
        spec:
          affinity:
            podAntiAffinity:
              preferredDuringSchedulingIgnoredDuringExecution:
              - weight: 100
                podAffinityTerm:
                  labelSelector:
                    matchExpressions:
                    - key: app
                      operator: In
                      values:
                      - pod-apollo-portal-server
                  topologyKey: kubernetes.io/hostname
          
          volumes:
            - name: volume-configmap-apollo-portal-server
              configMap:
                name: configmap-apollo-portal-server
                items:
                  - key: application-github.properties
                    path: application-github.properties
                  - key: apollo-env.properties
                    path: apollo-env.properties
          
          containers:
            - image: imageName:tag    # 更改为你的 docker registry 下的 image
              securityContext:
                privileged: true
              imagePullPolicy: IfNotPresent
              name: container-apollo-portal-server
              ports:
                - protocol: TCP
                  containerPort: 28070
              resources:
                limits:
                  memory: 512Mi          
              volumeMounts:
                - name: volume-configmap-apollo-portal-server
                  mountPath: /apollo-portal-server/config/application-github.properties
                  subPath: application-github.properties
                - name: volume-configmap-apollo-portal-server
                  mountPath: /apollo-portal-server/config/apollo-env.properties
                  subPath: apollo-env.properties
              
              env:
                - name: APOLLO_PORTAL_SERVICE_NAME
                  value: "service-apollo-portal-server.apollo"
              
              readinessProbe:
                tcpSocket:
                  port: 28070
                initialDelaySeconds: 10
                periodSeconds: 5
              
              livenessProbe:
                tcpSocket:
                  port: 28070
                # 120s 内, server 未启动则重启 container
                initialDelaySeconds: 120
                periodSeconds: 15
              
          dnsPolicy: ClusterFirst
          restartPolicy: Always

## 十二、启动服务 ##

分别启动configserver、adminserver、portalserver【必须严格按照此顺序启动，且在上一个服务启动成功后再启动下一个服务】， 命令如下:

> kubectl create -f apollo-config-server.yml  
> kubectl create -f apollo-admin-server.yml  
> kubectl create -f apollo-portal-server.yml

启动完成后可运行以下命令检查pod启动情况

> kubectl get pods -n apollo

结果如下，所有pod都为ready/running，则部署成功  
![pods][]  
查看某个pod日志，先进入容器

> kubectl exec -ti deployment-apollo-portal-server-59479cfddc-xf4fd /bin/sh -n apollo

再进入～/opt/logs目录tail对应log文件

## 十三、访问Apollo-portal服务 ##

k8s任意节点ip:30001访问，可在管理员工具-系统信息中查看服务状态  
![服务状态][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L20wXzM3MDYzNzg1_size_16_color_FFFFFF_t_70]  
到此，部署成功！

[Apollo-git]: https://github.com/ctripcorp/apollo
[pods]: https://img-blog.csdnimg.cn/20191217172415107.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L20wXzM3MDYzNzg1,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L20wXzM3MDYzNzg1_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20191217172631655.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L20wXzM3MDYzNzg1,size_16,color_FFFFFF,t_70