Skywalking-告警功能实践

超、凢脫俗 2021-06-10 20:40 1085阅读 0赞

*  方式：WebHook
 *  配置文件位置：\\config\\alarm-settings.yml
 *  建议不要使用 endpoint 规则，相比 service、instance 规则耗费更多内存及资源
 *  alarm参考：[https://github.com/apache/skywalking/blob/master/docs/en/setup/backend/backend-alarm.md][https_github.com_apache_skywalking_blob_master_docs_en_setup_backend_backend-alarm.md]
 *  metrics参考：[https://github.com/apache/skywalking/blob/master/docs/en/concepts-and-designs/oal.md][https_github.com_apache_skywalking_blob_master_docs_en_concepts-and-designs_oal.md]
 *  oal-scope参考：[https://github.com/apache/skywalking/blob/master/docs/en/concepts-and-designs/scope-definitions.md][https_github.com_apache_skywalking_blob_master_docs_en_concepts-and-designs_scope-definitions.md]
 *  默认 oal：\\config\\oal\\core.oal，可自定义oal于该文件中，后在alarm-settings文件中使用
 *  example
    
     *  // Calculate the percent of response status is true, for each service.
     *  endpoint\_success = from(Endpoint.\*).filter(status == true).percent()
     *  // Calculate the sum of response code in \[404, 500, 503\], for each service.
     *  endpoint\_abnormal = from(Endpoint.\*).filter(responseCode in \[404, 500, 503\]).sum()

**以下是一个可参考的告警配置：**

# Sample alarm rules.
    rules:
      # Rule unique name, must be ended with `_rule`.
      service_resp_time_rule:
        metrics-name: service_resp_time
        op: ">"
        threshold: 500
        period: 10
        count: 1
        silence-period: 5
        message: Response time of service {name} is more than 1000ms in last 10 minutes.
      service_sla_rule:
        # Indicator value need to be long, double or int
        metrics-name: service_sla
        op: "<"
        threshold: 8000
        # The length of time to evaluate the metric
        period: 10
        # How many times after the metric match the condition, will trigger alarm
        count: 2
        # How many times of checks, the alarm keeps silence after alarm triggered, default as same as period.
        silence-period: 3
        message: Successful rate of service {name} is lower than 80% in last 10 minutes.
      service_p90_sla_rule:
        # Indicator value need to be long, double or int
        metrics-name: service_p90
        op: ">"
        threshold: 500
        period: 10
        count: 1
        silence-period: 5
        message: 90% response time of service {name} is lower than 1000ms in last 10 minutes
      service_instance_resp_time_rule:
        metrics-name: service_instance_resp_time
        op: ">"
        threshold: 500
        period: 10
        count: 1
        silence-period: 5
        message: Response time of service instance {name} is more than 1000ms in last 10 minutes.
      endpoint_avg_rule:
        metrics-name: endpoint_avg
        op: ">"
        threshold: 500
        period: 10
        count: 1
        silence-period: 5
        message: Response time of endpoint {name} is more than 1000ms in last 10 minutes.
    
    webhooks:
    #  - http://127.0.0.1/notify/
    #  - http://127.0.0.1/go-wechat/

Skywalking每隔一段时间根据收集到的链路追踪的数据和配置的告警规则（如服务响应时间、服务响应 时间百分比）等，判断如果达到阈值则发送相应的告警信息。发送告警信息是通过调用webhook接口完 成，具体的webhook接口可以使用者自行定义，从而开发者可以在指定的webhook接口中编写各种告 警方式，比如邮件、短信等。告警的信息也可以在RocketBot中查看到。

以下是默认的告警规则配置，位于skywalking安装目录下的conﬁg文件夹下 alarm-settings.yml文件 中：

rules:
      # Rule unique name, must be ended with `_rule`.
      service_resp_time_rule:
        metrics-name: service_resp_time
        op: ">"
        threshold: 1000
        period: 10
        count: 3
        silence-period: 5
        message: Response time of service {name} is more than 1000ms in 3 minutes of last 10 minutes.
      service_sla_rule:
        # Metrics value need to be long, double or int
        metrics-name: service_sla
        op: "<"
        threshold: 8000
        # The length of time to evaluate the metrics
        period: 10
        # How many times after the metrics match the condition, will trigger alarm
        count: 2
        # How many times of checks, the alarm keeps silence after alarm triggered, default as same as period.
        silence-period: 3
        message: Successful rate of service {name} is lower than 80% in 2 minutes of last 10 minutes
      service_p90_sla_rule:
        # Metrics value need to be long, double or int
        metrics-name: service_p90
        op: ">"
        threshold: 1000
        period: 10
        count: 3
        silence-period: 5
        message: 90% response time of service {name} is more than 1000ms in 3 minutes of last 10 minutes
      service_instance_resp_time_rule:
        metrics-name: service_instance_resp_time
        op: ">"
        threshold: 1000
        period: 10
        count: 2
        silence-period: 5
        message: Response time of service instance {name} is more than 1000ms in 2 minutes of last 10 minutes
    #  Active endpoint related metrics alarm will cost more memory than service and service instance metrics alarm.
    #  Because the number of endpoint is much more than service and instance.
    #
    #  endpoint_avg_rule:
    #    metrics-name: endpoint_avg
    #    op: ">"
    #    threshold: 1000
    #    period: 10
    #    count: 2
    #    silence-period: 5
    #    message: Response time of endpoint {name} is more than 1000ms in 2 minutes of last 10 minutes
    
    webhooks:
    #  - http://127.0.0.1/notify/
    #  - http://127.0.0.1/go-wechat/

**以上文件定义了默认的4种规则**

1.  最近3分钟内服务的平均响应时间超过1秒
2.  最近2分钟服务成功率低于80%
3.  最近3分钟90%服务响应时间超过1秒
4.  最近2分钟内服务实例的平均响应时间超过1秒 规则中的参数属性如下

**属性参照表**

<table> 
 <thead> 
  <tr> 
   <th>属性</th> 
   <th>含义</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>metrics-name</td> 
   <td>oal脚本中的度量名称</td> 
  </tr> 
  <tr> 
   <td>threshold</td> 
   <td>阈值，与metrics-name和下面的比较符号相匹配</td> 
  </tr> 
  <tr> 
   <td>op</td> 
   <td>比较操作符，可以设定&gt;,&lt;,=</td> 
  </tr> 
  <tr> 
   <td>period</td> 
   <td>多久检查一次当前的指标数据是否符合告警规则，单位分钟</td> 
  </tr> 
  <tr> 
   <td>count</td> 
   <td>达到多少次后，发送告警消息</td> 
  </tr> 
  <tr> 
   <td>silence-period</td> 
   <td>在多久之内，忽略相同的告警消息</td> 
  </tr> 
  <tr> 
   <td>message</td> 
   <td>告警消息内容</td> 
  </tr> 
  <tr> 
   <td>include-names</td> 
   <td>本规则告警生效的服务列表</td> 
  </tr> 
 </tbody> 
</table>

> webhooks可以配置告警产生时的调用地址。

### 3.4.2 告警功能测试代码 ###

编写告警功能接口来进行测试，创建skywalking\_alarm项目。

**AlarmController**

import org.springframework.web.bind.annotation.GetMapping;
    import org.springframework.web.bind.annotation.RestController;
    
    @RestController
    public class AlarmController {
    
        //每次调用睡眠1.5秒，模拟超时的报警
        @GetMapping("/timeout")
        public String timeout(){
            try {
                Thread.sleep(1500);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
    
            return "timeout";
        }
    }

该接口主要用于模拟超时，多次调用之后就可以生成告警信息。

**WebHooks**

import com.sf.saas.skywalking_alarm.pojo.AlarmMessage;
    import org.springframework.web.bind.annotation.GetMapping;
    import org.springframework.web.bind.annotation.PostMapping;
    import org.springframework.web.bind.annotation.RequestBody;
    import org.springframework.web.bind.annotation.RestController;
    
    import java.util.ArrayList;
    import java.util.List;
    
    @RestController
    public class WebHooks {
    
        private List<AlarmMessage> lastList = new ArrayList<>();
    
        @PostMapping("/webhook")
        public void  webhook(@RequestBody List<AlarmMessage> alarmMessageList){
            lastList = alarmMessageList;
        }
    
        @GetMapping("/show")
        public List<AlarmMessage> show(){
            return lastList;
        }
    }

> 产生告警时会调用webhook接口，该接口必须是Post类型，同时接口参数使用RequestBody。参 数格式为：

[{
    	"scopeId": 1,
    	"scope": "SERVICE",
    	"name": "serviceA",
    	"id0": 12,
    	"id1": 0,
    	"ruleName": "service_resp_time_rule",
    	"alarmMessage": "alarmMessage xxxx",
    	"startTime": 1560524171000
    }, {
    	"scopeId": 1,
    	"scope": "SERVICE",
    	"name": "serviceB",
    	"id0": 23,
    	"id1": 0,
    	"ruleName": "service_resp_time_rule",
    	"alarmMessage": "alarmMessage yyy",
    	"startTime": 1560524171000
    }]

**AlarmMessage**

public class AlarmMessage {
        private int scopeId;
        private String name;
        private int id0;
        private int id1;
        //告警的消息
        private String alarmMessage;
        //告警的产生时间
        private long startTime;
    
        public int getScopeId() {
            return scopeId;
        }
    
        public void setScopeId(int scopeId) {
            this.scopeId = scopeId;
        }
    
        public String getName() {
            return name;
        }
    
        public void setName(String name) {
            this.name = name;
        }
    
        public int getId0() {
            return id0;
        }
    
        public void setId0(int id0) {
            this.id0 = id0;
        }
    
        public int getId1() {
            return id1;
        }
    
        public void setId1(int id1) {
            this.id1 = id1;
        }
    
        public String getAlarmMessage() {
            return alarmMessage;
        }
    
        public void setAlarmMessage(String alarmMessage) {
            this.alarmMessage = alarmMessage;
        }
    
        public long getStartTime() {
            return startTime;
        }
    
        public void setStartTime(long startTime) {
            this.startTime = startTime;
        }
    
        @Override
        public String toString() {
            return "AlarmMessage{" +
                    "scopeId=" + scopeId +
                    ", name='" + name + '\'' +
                    ", id0=" + id0 +
                    ", id1=" + id1 +
                    ", alarmMessage='" + alarmMessage + '\'' +
                    ", startTime=" + startTime +
                    '}';
        }
    }

实体类用于接口告警信息

### 3.4.3 部署测试 ###

首先需要修改告警规则配置文件，将webhook地址修改为

webhooks: 
      - http://127.0.0.1:8089/webhook

然后重启skywalking  
1、将 skywalking\_alarm.jar上传至 /usr/local/skywalking目录下。

2、启动skywalking\_alarm应用，等待启动成功。

java -javaagent:/usr/local/skywalking/apache-skywalking-apm- 
    bin/agent/skywalking-agent.jar -Dskywalking.agent.service_name=skywalking_alarm -jar skywalking_alarm.jar

3、不停调用接口，接口地址为：http://虚拟机IP:8089/timeout

4、直到出现告警：

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjUyODI2Ng_size_16_color_FFFFFF_t_70]

5、查看告警信息接口：http://虚拟机IP:8089/show  
![在这里插入图片描述][20200806153848112.jpg]

从上图中可以看到，我们已经获取到了告警相关的信息，在生产中使用可以在webhook接口中对接短 信、邮件等平台，当告警出现时能迅速发送信息给对应的处理人员，提高故障处理的速度。

[https_github.com_apache_skywalking_blob_master_docs_en_setup_backend_backend-alarm.md]: https://github.com/apache/skywalking/blob/master/docs/en/setup/backend/backend-alarm.md
[https_github.com_apache_skywalking_blob_master_docs_en_concepts-and-designs_oal.md]: https://github.com/apache/skywalking/blob/master/docs/en/concepts-and-designs/oal.md
[https_github.com_apache_skywalking_blob_master_docs_en_concepts-and-designs_scope-definitions.md]: https://github.com/apache/skywalking/blob/master/docs/en/concepts-and-designs/scope-definitions.md
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjUyODI2Ng_size_16_color_FFFFFF_t_70]: /images/20210527/26b4f266bb0b485dbbcde991cedd7038.png
[20200806153848112.jpg]: /images/20210527/1b62e2a13c594372ab5a7aee56cb254e.png