【Docker学习】14、使用 Docker 搭建 Prometheus 监控告警系统（邮件通知）

柔情只为你懂 2022-12-16 06:19 801阅读 0赞

### 文章目录 ###

*  1、Prometheus 监控告警系统
 *   *   *  （1）Alertmanager
         *  （2）Alertmanager 环境搭建
         *  （3）Prometheus 配置 Alertmanager
         *  （4）测试邮件发送以及接收
 *  微信公众号

# 1、Prometheus 监控告警系统 #

### （1）Alertmanager ###

Alertmanager处理客户端应用程序(如Prometheus服务器)发送的警报。它负责重复数据删除、分组和路由它们到正确的接收端集成，如电子邮件、PagerDuty或OpsGenie。它还负责警报的沉默和抑制

**分组**

分组将类似性质的警报分类为单个通知。这在较大的中断中特别有用，当大量系统一次出现故障并且可能同时触发数百到数千个警报时。

示例：发生网络分区时，群集中正在运行数十个或数百个服务实例。您有一半的服务实例不再可以访问数据库。Prometheus中的警报规则配置为在每个服务实例无法与数据库通信时为其发送警报。结果，数百个警报被发送到Alertmanager。

作为用户，人们只希望获得一个页面，同时仍然能够准确查看受影响的服务实例。因此，可以将Alertmanager配置为按警报的群集和警报名称分组警报，以便它发送一个紧凑的通知。

警报的分组，分组通知的时间以及这些通知的接收者由配置文件中的路由树配置。

**抑制**

禁止是一种概念，如果某些其他警报已经触发，则抑制某些警报的通知。

示例：正在触发警报，通知您整个群集无法访问。可以将Alertmanager配置为使与该群集有关的所有其他警报静音。这样可以防止与实际问题无关的数百或数千个触发警报的通知。

通过Alertmanager的配置文件配置禁止。

**沉默**

静默是一种简单的方法，可以在给定时间内简单地使警报静音。沉默是根据匹配器配置的，就像路由树一样。检查传入警报是否与活动静默的所有相等或正则表达式匹配项匹配。如果这样做，则不会针对该警报发送任何通知。

在Alertmanager的Web界面中配置沉默。

*  [Github alertmanager][]

### （2）Alertmanager 环境搭建 ###

拉取 Prometheus 提供的监控告警镜像

docker pull prom/alertmanager

在挂载的目录下添加 alertmanager 配置文件 alertmanager.yml

# 全局参数
    global:
      smtp_smarthost: 'smtp.qq.com:465'
      smtp_from: '3210054449@qq.com'
      smtp_auth_username: '3210054449@qq.com'
      smtp_auth_password: '********'
      smtp_require_tls: false #要设置后才能发送成功，默认是true
    
    route:
      group_interval: 1m
      repeat_interval: 1m
      receiver: 'mail-receiver'
    # 接收者
    receivers:
      - name: 'mail-receiver'
        email_configs:
        - to: '3210054449@qq.com'
    # 告警抑制
    inhibit_rules:
      - source_match:
          severity: 'critical'
        target_match:
          severity: 'warning'
        equal: ['alertname', 'dev', 'instance']

构建 alertmanager 容器

docker run -d -p 9093:9093 \
    -v /tellsea/docker/alertmanager/:/etc/alertmanager/ \
    --name=alertmanager prom/alertmanager

启动 alertmanager 容器

docker start alertmanager

浏览器访问

http://101.200.127.67:9093/#/alerts

访问效果如下

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4NzYyMjM3_size_16_color_FFFFFF_t_70_pic_center]

### （3）Prometheus 配置 Alertmanager ###

在 Prometheus 挂载的目录下添加报警规则的配置文件

*  创建 node-exporter-record-rule.yml

groups:
      - name: node-exporter-record
        rules:
        - expr: up{ job=~"node-exporter"}
          record: node_exporter:up
          labels:
            desc: "节点是否在线, 在线1,不在线0"
            unit: " "
            job: "node-exporter"
        - expr: time() - node_boot_time_seconds{ }
          record: node_exporter:node_uptime
          labels:
            desc: "节点的运行时间"
            unit: "s"
            job: "node-exporter"
    ##############################################################################################
    # cpu #
        - expr: (1 - avg by (environment,instance) (irate(node_cpu_seconds_total{ job="node-exporter",mode="idle"}[5m])))  * 100
          record: node_exporter:cpu:total:percent
          labels:
            desc: "节点的cpu总消耗百分比"
            unit: "%"
            job: "node-exporter"
    
        - expr: (avg by (environment,instance) (irate(node_cpu_seconds_total{ job="node-exporter",mode="idle"}[5m])))  * 100
          record: node_exporter:cpu:idle:percent
          labels:
            desc: "节点的cpu idle百分比"
            unit: "%"
            job: "node-exporter"
    
        - expr: (avg by (environment,instance) (irate(node_cpu_seconds_total{ job="node-exporter",mode="iowait"}[5m])))  * 100
          record: node_exporter:cpu:iowait:percent
          labels:
            desc: "节点的cpu iowait百分比"
            unit: "%"
            job: "node-exporter"
    
    
        - expr: (avg by (environment,instance) (irate(node_cpu_seconds_total{ job="node-exporter",mode="system"}[5m])))  * 100
          record: node_exporter:cpu:system:percent
          labels:
            desc: "节点的cpu system百分比"
            unit: "%"
            job: "node-exporter"
    
        - expr: (avg by (environment,instance) (irate(node_cpu_seconds_total{ job="node-exporter",mode="user"}[5m])))  * 100
          record: node_exporter:cpu:user:percent
          labels:
            desc: "节点的cpu user百分比"
            unit: "%"
            job: "node-exporter"
    
        - expr: (avg by (environment,instance) (irate(node_cpu_seconds_total{ job="node-exporter",mode=~"softirq|nice|irq|steal"}[5m])))  * 100
          record: node_exporter:cpu:other:percent
          labels:
            desc: "节点的cpu 其他的百分比"
            unit: "%"
            job: "node-exporter"
    ##############################################################################################
    
    
    ##############################################################################################
    # memory #
        - expr: node_memory_MemTotal_bytes{ job="node-exporter"}
          record: node_exporter:memory:total
          labels:
            desc: "节点的内存总量"
            unit: byte
            job: "node-exporter"
    
        - expr: node_memory_MemFree_bytes{ job="node-exporter"}
          record: node_exporter:memory:free
          labels:
            desc: "节点的剩余内存量"
            unit: byte
            job: "node-exporter"
    
        - expr: node_memory_MemTotal_bytes{ job="node-exporter"} - node_memory_MemFree_bytes{ job="node-exporter"}
          record: node_exporter:memory:used
          labels:
            desc: "节点的已使用内存量"
            unit: byte
            job: "node-exporter"
    
        - expr: node_memory_MemTotal_bytes{ job="node-exporter"} - node_memory_MemAvailable_bytes{ job="node-exporter"}
          record: node_exporter:memory:actualused
          labels:
            desc: "节点用户实际使用的内存量"
            unit: byte
            job: "node-exporter"
    
        - expr: (1-(node_memory_MemAvailable_bytes{ job="node-exporter"} / (node_memory_MemTotal_bytes{ job="node-exporter"})))* 100
          record: node_exporter:memory:used:percent
          labels:
            desc: "节点的内存使用百分比"
            unit: "%"
            job: "node-exporter"
    
        - expr: ((node_memory_MemAvailable_bytes{ job="node-exporter"} / (node_memory_MemTotal_bytes{ job="node-exporter"})))* 100
          record: node_exporter:memory:free:percent
          labels:
            desc: "节点的内存剩余百分比"
            unit: "%"
            job: "node-exporter"
    ##############################################################################################
    # load #
        - expr: sum by (instance) (node_load1{ job="node-exporter"})
          record: node_exporter:load:load1
          labels:
            desc: "系统1分钟负载"
            unit: " "
            job: "node-exporter"
    
        - expr: sum by (instance) (node_load5{ job="node-exporter"})
          record: node_exporter:load:load5
          labels:
            desc: "系统5分钟负载"
            unit: " "
            job: "node-exporter"
    
        - expr: sum by (instance) (node_load15{ job="node-exporter"})
          record: node_exporter:load:load15
          labels:
            desc: "系统15分钟负载"
            unit: " "
            job: "node-exporter"
    
    ##############################################################################################
    # disk #
        - expr: node_filesystem_size_bytes{ job="node-exporter" ,fstype=~"ext4|xfs"}
          record: node_exporter:disk:usage:total
          labels:
            desc: "节点的磁盘总量"
            unit: byte
            job: "node-exporter"
    
        - expr: node_filesystem_avail_bytes{ job="node-exporter",fstype=~"ext4|xfs"}
          record: node_exporter:disk:usage:free
          labels:
            desc: "节点的磁盘剩余空间"
            unit: byte
            job: "node-exporter"
    
        - expr: node_filesystem_size_bytes{ job="node-exporter",fstype=~"ext4|xfs"} - node_filesystem_avail_bytes{ job="node-exporter",fstype=~"ext4|xfs"}
          record: node_exporter:disk:usage:used
          labels:
            desc: "节点的磁盘使用的空间"
            unit: byte
            job: "node-exporter"
    
        - expr:  (1 - node_filesystem_avail_bytes{ job="node-exporter",fstype=~"ext4|xfs"} / node_filesystem_size_bytes{ job="node-exporter",fstype=~"ext4|xfs"}) * 100
          record: node_exporter:disk:used:percent
          labels:
            desc: "节点的磁盘的使用百分比"
            unit: "%"
            job: "node-exporter"
    
        - expr: irate(node_disk_reads_completed_total{ job="node-exporter"}[1m])
          record: node_exporter:disk:read:count:rate
          labels:
            desc: "节点的磁盘读取速率"
            unit: "次/秒"
            job: "node-exporter"
    
        - expr: irate(node_disk_writes_completed_total{ job="node-exporter"}[1m])
          record: node_exporter:disk:write:count:rate
          labels:
            desc: "节点的磁盘写入速率"
            unit: "次/秒"
            job: "node-exporter"
    
        - expr: (irate(node_disk_written_bytes_total{ job="node-exporter"}[1m]))/1024/1024
          record: node_exporter:disk:read:mb:rate
          labels:
            desc: "节点的设备读取MB速率"
            unit: "MB/s"
            job: "node-exporter"
    
        - expr: (irate(node_disk_read_bytes_total{ job="node-exporter"}[1m]))/1024/1024
          record: node_exporter:disk:write:mb:rate
          labels:
            desc: "节点的设备写入MB速率"
            unit: "MB/s"
            job: "node-exporter"
    
    ##############################################################################################
    # filesystem #
        - expr:   (1 -node_filesystem_files_free{ job="node-exporter",fstype=~"ext4|xfs"} / node_filesystem_files{ job="node-exporter",fstype=~"ext4|xfs"}) * 100
          record: node_exporter:filesystem:used:percent
          labels:
            desc: "节点的inode的剩余可用的百分比"
            unit: "%"
            job: "node-exporter"
    #############################################################################################
    # filefd #
        - expr: node_filefd_allocated{ job="node-exporter"}
          record: node_exporter:filefd_allocated:count
          labels:
            desc: "节点的文件描述符打开个数"
            unit: "%"
            job: "node-exporter"
    
        - expr: node_filefd_allocated{ job="node-exporter"}/node_filefd_maximum{ job="node-exporter"} * 100
          record: node_exporter:filefd_allocated:percent
          labels:
            desc: "节点的文件描述符打开百分比"
            unit: "%"
            job: "node-exporter"
    
    #############################################################################################
    # network #
        - expr: avg by (environment,instance,device) (irate(node_network_receive_bytes_total{ device=~"eth0|eth1|ens33|ens37"}[1m]))
          record: node_exporter:network:netin:bit:rate
          labels:
            desc: "节点网卡eth0每秒接收的比特数"
            unit: "bit/s"
            job: "node-exporter"
    
        - expr: avg by (environment,instance,device) (irate(node_network_transmit_bytes_total{ device=~"eth0|eth1|ens33|ens37"}[1m]))
          record: node_exporter:network:netout:bit:rate
          labels:
            desc: "节点网卡eth0每秒发送的比特数"
            unit: "bit/s"
            job: "node-exporter"
    
        - expr: avg by (environment,instance,device) (irate(node_network_receive_packets_total{ device=~"eth0|eth1|ens33|ens37"}[1m]))
          record: node_exporter:network:netin:packet:rate
          labels:
            desc: "节点网卡每秒接收的数据包个数"
            unit: "个/秒"
            job: "node-exporter"
    
        - expr: avg by (environment,instance,device) (irate(node_network_transmit_packets_total{ device=~"eth0|eth1|ens33|ens37"}[1m]))
          record: node_exporter:network:netout:packet:rate
          labels:
            desc: "节点网卡发送的数据包个数"
            unit: "个/秒"
            job: "node-exporter"
    
        - expr: avg by (environment,instance,device) (irate(node_network_receive_errs_total{ device=~"eth0|eth1|ens33|ens37"}[1m]))
          record: node_exporter:network:netin:error:rate
          labels:
            desc: "节点设备驱动器检测到的接收错误包的数量"
            unit: "个/秒"
            job: "node-exporter"
    
        - expr: avg by (environment,instance,device) (irate(node_network_transmit_errs_total{ device=~"eth0|eth1|ens33|ens37"}[1m]))
          record: node_exporter:network:netout:error:rate
          labels:
            desc: "节点设备驱动器检测到的发送错误包的数量"
            unit: "个/秒"
            job: "node-exporter"
    
        - expr: node_tcp_connection_states{ job="node-exporter", state="established"}
          record: node_exporter:network:tcp:established:count
          labels:
            desc: "节点当前established的个数"
            unit: "个"
            job: "node-exporter"
    
        - expr: node_tcp_connection_states{ job="node-exporter", state="time_wait"}
          record: node_exporter:network:tcp:timewait:count
          labels:
            desc: "节点timewait的连接数"
            unit: "个"
            job: "node-exporter"
    
        - expr: sum by (environment,instance) (node_tcp_connection_states{ job="node-exporter"})
          record: node_exporter:network:tcp:total:count
          labels:
            desc: "节点tcp连接总数"
            unit: "个"
            job: "node-exporter"
    
    #############################################################################################
    # process #
        - expr: node_processes_state{ state="Z"}
          record: node_exporter:process:zoom:total:count
          labels:
            desc: "节点当前状态为zoom的个数"
            unit: "个"
            job: "node-exporter"
    #############################################################################################
    # other #
        - expr: abs(node_timex_offset_seconds{ job="node-exporter"})
          record: node_exporter:time:offset
          labels:
            desc: "节点的时间偏差"
            unit: "s"
            job: "node-exporter"
    
    #############################################################################################
    
        - expr: count by (instance) ( count by (instance,cpu) (node_cpu_seconds_total{  mode='system'}) )
          record: node_exporter:cpu:count

*  创建 node-exporter-alert-rule.yml

groups:
      - name: node-exporter-alert
        rules:
        - alert: node-exporter-down
          expr: node_exporter:up == 0
          for: 1m
          labels:
            severity: 'critical'
          annotations:
            summary: "instance: { {  $labels.instance }} 宕机了"
            description: "instance: { {  $labels.instance }} \n- job: { {  $labels.job }} 关机了， 时间已经1分钟了。"
            value: "{ { $value }}"
            instance: "{ { $labels.instance }}"
    
    
    
        - alert: node-exporter-cpu-high
          expr:  node_exporter:cpu:total:percent > 80
          for: 3m
          labels:
            severity: info
          annotations:
            summary: "instance: { {  $labels.instance }} cpu 使用率高于 { {  $value }}"
            description: "instance: { {  $labels.instance }} \n- job: { {  $labels.job }} CPU使用率已经持续三分钟高过80% 。"
            value: "{ { $value }}"
            instance: "{ { $labels.instance }}"
    
        - alert: node-exporter-cpu-iowait-high
          expr:  node_exporter:cpu:iowait:percent >= 12
          for: 3m
          labels:
            severity: info
          annotations:
            summary: "instance: { {  $labels.instance }} cpu iowait 使用率高于 { {  $value }}"
            description: "instance: { {  $labels.instance }} \n- job: { {  $labels.job }} cpu iowait使用率已经持续三分钟高过12%"
            value: "{ { $value }}"
            instance: "{ { $labels.instance }}"
    
    
        - alert: node-exporter-load-load1-high
          expr:  (node_exporter:load:load1) > (node_exporter:cpu:count) * 1.2
          for: 3m
          labels:
            severity: info
          annotations:
            summary: "instance: { {  $labels.instance }} load1 使用率高于 { {  $value }}"
            description: ""
            value: "{ { $value }}"
            instance: "{ { $labels.instance }}"
    
    
        - alert: node-exporter-memory-high
          expr:  node_exporter:memory:used:percent > 85
          for: 3m
          labels:
            severity: info
          annotations:
            summary: "instance: { {  $labels.instance }} memory 使用率高于 { {  $value }}"
            description: ""
            value: "{ { $value }}"
            instance: "{ { $labels.instance }}"
    
    
        - alert: node-exporter-disk-high
          expr:  node_exporter:disk:used:percent > 88
          for: 10m
          labels:
            severity: info
          annotations:
            summary: "instance: { {  $labels.instance }} disk 使用率高于 { {  $value }}"
            description: ""
            value: "{ { $value }}"
            instance: "{ { $labels.instance }}"
    
    
        - alert: node-exporter-disk-read:count-high
          expr:  node_exporter:disk:read:count:rate > 3000
          for: 2m
          labels:
            severity: info
          annotations:
            summary: "instance: { {  $labels.instance }} iops read 使用率高于 { {  $value }}"
            description: ""
            value: "{ { $value }}"
            instance: "{ { $labels.instance }}"
    
    
        - alert: node-exporter-disk-write-count-high
          expr:  node_exporter:disk:write:count:rate > 3000
          for: 2m
          labels:
            severity: info
          annotations:
            summary: "instance: { {  $labels.instance }} iops write 使用率高于 { {  $value }}"
            description: ""
            value: "{ { $value }}"
            instance: "{ { $labels.instance }}"
    
    
    
    
    
        - alert: node-exporter-disk-read-mb-high
          expr:  node_exporter:disk:read:mb:rate > 60
          for: 2m
          labels:
            severity: info
          annotations:
            summary: "instance: { {  $labels.instance }} 读取字节数 高于 { {  $value }}"
            description: ""
            instance: "{ { $labels.instance }}"
            value: "{ { $value }}"
    
    
        - alert: node-exporter-disk-write-mb-high
          expr:  node_exporter:disk:write:mb:rate > 60
          for: 2m
          labels:
            severity: info
          annotations:
            summary: "instance: { {  $labels.instance }} 写入字节数 高于 { {  $value }}"
            description: ""
            value: "{ { $value }}"
            instance: "{ { $labels.instance }}"
    
    
        - alert: node-exporter-filefd-allocated-percent-high
          expr:  node_exporter:filefd_allocated:percent > 80
          for: 10m
          labels:
            severity: info
          annotations:
            summary: "instance: { {  $labels.instance }} 打开文件描述符 高于 { {  $value }}"
            description: ""
            value: "{ { $value }}"
            instance: "{ { $labels.instance }}"
    
    
        - alert: node-exporter-network-netin-error-rate-high
          expr:  node_exporter:network:netin:error:rate > 4
          for: 1m
          labels:
            severity: info
          annotations:
            summary: "instance: { {  $labels.instance }} 包进入的错误速率 高于 { {  $value }}"
            description: ""
            value: "{ { $value }}"
            instance: "{ { $labels.instance }}"
    
        - alert: node-exporter-network-netin-packet-rate-high
          expr:  node_exporter:network:netin:packet:rate > 35000
          for: 1m
          labels:
            severity: info
          annotations:
            summary: "instance: { {  $labels.instance }} 包进入速率 高于 { {  $value }}"
            description: ""
            value: "{ { $value }}"
            instance: "{ { $labels.instance }}"
    
    
        - alert: node-exporter-network-netout-packet-rate-high
          expr:  node_exporter:network:netout:packet:rate > 35000
          for: 1m
          labels:
            severity: info
          annotations:
            summary: "instance: { {  $labels.instance }} 包流出速率 高于 { {  $value }}"
            description: ""
            value: "{ { $value }}"
            instance: "{ { $labels.instance }}"
    
    
        - alert: node-exporter-network-tcp-total-count-high
          expr:  node_exporter:network:tcp:total:count > 40000
          for: 1m
          labels:
            severity: info
          annotations:
            summary: "instance: { {  $labels.instance }} tcp连接数量 高于 { {  $value }}"
            description: ""
            value: "{ { $value }}"
            instance: "{ { $labels.instance }}"
    
    
        - alert: node-exporter-process-zoom-total-count-high
          expr:  node_exporter:process:zoom:total:count > 10
          for: 10m
          labels:
            severity: info
          annotations:
            summary: "instance: { {  $labels.instance }} 僵死进程数量 高于 { {  $value }}"
            description: ""
            value: "{ { $value }}"
            instance: "{ { $labels.instance }}"
    
    
        - alert: node-exporter-time-offset-high
          expr:  node_exporter:time:offset > 0.03
          for: 2m
          labels:
            severity: info
          annotations:
            summary: "instance: { {  $labels.instance }} { {  $labels.desc }}  { {  $value }} { {  $labels.unit }}"
            description: ""
            value: "{ { $value }}"
            instance: "{ { $labels.instance }}"

修改 Prometheus 配置文件，增加报警信息

rule_files:
      # 通配规则文件
      - "*rule.yml"
    # Alertmanager 配置
    alerting:
      alertmanagers:
      - scheme: http
      - static_configs:
        - targets: ["101.200.127.67:9093"]

重启一下Prometheus

docker restart prometheus

查看报警信息

http://101.200.127.67:9090/alerts

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4NzYyMjM3_size_16_color_FFFFFF_t_70_pic_center 1]

查看规则文件配置

http://101.200.127.67:9090/rules

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4NzYyMjM3_size_16_color_FFFFFF_t_70_pic_center 2]

### （4）测试邮件发送以及接收 ###

触发邮件通知只需要满足规则制定的条件即可，这里我直接演示关闭一个组件

docker stop node-exporter

然后过一会儿访问 alertmananger 发现多了点信息

http://101.200.127.67:9093/#/alerts

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4NzYyMjM3_size_16_color_FFFFFF_t_70_pic_center 3]

Prometheus 中也能看到相应的错误信息

http://101.200.127.67:9090/alerts

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4NzYyMjM3_size_16_color_FFFFFF_t_70_pic_center 4]  
然后过了一会儿（时间久一点），收到QQ邮件

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4NzYyMjM3_size_16_color_FFFFFF_t_70_pic_center 5]

# 微信公众号 #

![每天Get一个小技巧][Get]

[Github alertmanager]: https://github.com/prometheus/alertmanager
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4NzYyMjM3_size_16_color_FFFFFF_t_70_pic_center]: /images/20221123/05bbd74450614dd8b5a315b4f6a72cfe.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4NzYyMjM3_size_16_color_FFFFFF_t_70_pic_center 1]: /images/20221123/8e1379b23c504063827ae11948b22dd2.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4NzYyMjM3_size_16_color_FFFFFF_t_70_pic_center 2]: /images/20221123/cb5d541354ea44bf8a1e4c7eba94bc01.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4NzYyMjM3_size_16_color_FFFFFF_t_70_pic_center 3]: /images/20221123/aae3fc553c9a4edcab1d91836e8c92d4.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4NzYyMjM3_size_16_color_FFFFFF_t_70_pic_center 4]: /images/20221123/b9eb27ed82b549478271f47c08f582b7.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4NzYyMjM3_size_16_color_FFFFFF_t_70_pic_center 5]: /images/20221123/f20a26de32154f9783b511f595cde628.png
[Get]: /images/20221123/e30aeb272e2342cebe95034548c68294.png