ES相关概念理解,常用查询

秒速五厘米 2023-07-18 09:25 67阅读 0赞

## Query 与 Filter ##

1、query是要相关性评分的，filter不要  
2、query结果无法缓存，filter可以（空间满，使用LRU淘汰）

全文搜索、评分排序，使用query  
是非过滤、精确匹配，使用filter

GET /nginx-elk-2020.03.22/_search
    {
      "query": {
        "bool":{
          "must": [
            {
             "match": {
                "http_host":"product.mlalgo.api"
              }
            }
          ], "filter": {
            "range": {
              "@timestamp": {
                "from": "2020-03-22T10:00:52.000Z",
                "to": "2020-03-22T18:00:52.000Z"
              }
            }
          }
        }
      }
    }

## bool 过滤 ##

> bool 过滤可以用来合并多个过滤条件查询结果的布尔逻辑，它包含一下操作符：  
> must :: 多个查询条件的完全匹配,相当于 and 。  
> must\_not :: 多个查询条件的相反匹配，相当于 not 。  
> should :: 至少有一个查询条件匹配, 相当于 or 。

## multi\_match 查询 ##

GET /nginx-elk-2020.03.22/_search
    {
      "size": 10, 
      "query": {
       "multi_match": {
         "query": "product",
         "fields": ["http_host", "request_uri"]
       }
      }
    }

## 使用 from 及 size 参数进行分页 ##

GET /nginx-elk-2020.03.24/_search
    {
      "from": 30,
      "size":10
    }

## terms 过滤 ##

terms 跟 term 有点类似，但 terms 允许指定多个匹配条件

GET /nginx-elk-2020.03.22/_search
    {
      "size": 10, 
      "query": {
        "bool": {
          "must": [
            {
              "terms": {
              "http_host": ["product.mlalgo.api", "api"]
              }
            }
            ]
        }
      }
    }

## 排序 ##

GET /nginx-elk-2020.03.22/_search
    {
      "size": 10, 
      "query": {
        "bool": {
          "filter": {
            "term": {
              "http_host": "product.mlalgo.api"
            }
          }
        }
      }, "sort": [
        {
          "upstream_response_time": {
            "order": "desc"
          }
        }
      ]
    }

## 多级排序 ##

## aggregation ##

聚合查看时间分布

GET /nginx-elk-2020.03.24/_search
    {
     "size": 10
     , "query": {
       "bool": {
         "filter": {
           "term": {
             "http_host": "product.mlalgo.api"
           }
         }
       }
     }
     , "aggs": {
       "percentile_over_time": {
         "date_histogram": {
           "field": "@timestamp",
           "interval": "hour"
         }, "aggs": {
           "count_persent": {
             "percentiles": {
               "field": "upstream_response_time",
               "percents": [
                 1,
                 5,
                 25,
                 50,
                 75,
                 95,
                 99
               ]
             }
           }
         }
       }
     }
    }

聚合查看各项平均指标

GET /nginx-elk-2020.03.22/_search
    {
      "size": 10, 
      "query": {
        "bool": {
          "must": [
            {
              "term": {
              "http_host": "product.mlalgo.api"
            }},
            {"term": {
              "status": "200"
            }},
            {
              "range": {
                "@timestamp": {
                  "from": "2020-03-22T10:00:52.000Z",
                  "to": "2020-03-22T18:00:52.000Z"
                }
              }
            }
          ]
          , "filter": {
            "range": {
                "upstream_response_time": {
                  "gte": 0.05,
                  "lte": 1
                }
              }
          }
        }
      }
      , "aggs": {
        "rt_stats": {
          "extended_stats": {
            "field": "upstream_response_time"
          }
        }
      }
    }

## pipeline aggregation ##

在已有aggregation返回数组数据之后，再对这组数值值做一次运算。比如对响应时间设置如下：周期为7，移动窗口为30，alpha、beta、gamma参数为0.5，holt-winter季节性预测2个未来值：

GET /nginx-elk-2020.03.24/_search
    {
     "size": 10
     , "query": {
       "bool": {
         "filter": {
           "term": {
             "http_host": "product.mlalgo.api"
           }
         }
       }
     }
     , "aggs": {
       "response_histogram": {
         "date_histogram": {
           "field": "@timestamp",
           "interval": "hour"
         }, "aggs": {
           "avg_response": {
             "avg": {
               "field": "upstream_response_time"
             }
           },
           "time_movavg":{
             "moving_avg": {
               "buckets_path": "avg_response",
               "window": 30,
               "model": "holt_winters",
               "predict":2,
               "settings": {
                 "type": "mult",
                 "alpha": 0.5,
                 "beta": 0.5,
                 "gamma": 0.5
               }
             }
           }
         }
       }
     }
    }

## 桶排序：aggregation sort ##

排序时用“.”表达桶之间的嵌套关系

GET /nginx-elk-2020.03.22/_search
    {
      "size": 10, 
      "query": {
        "bool": {
          "must": [
            {
              "term": {
              "http_host": "product.mlalgo.api"
            }},
            {"term": {
              "status": "200"
            }},
            {
              "range": {
                "@timestamp": {
                  "from": "2020-03-22T10:00:52.000Z",
                  "to": "2020-03-22T18:00:52.000Z"
                }
              }
            }
          ]
          , "filter": {
            "range": {
                "upstream_response_time": {
                  "gte": 0.05,
                  "lte": 1
                }
              }
          }
        }
      }, "aggs": {
        "date_histo": {
          "date_histogram": {
            "field": "@timestamp",
            "order": {
              "rt_stats.max": "desc"
            }, 
            "interval": "hour"
          }, "aggs": {
            "rt_stats": {
              "extended_stats": {
                "field": "upstream_response_time"
              }
            }
          }
        }
      }
    }

## field\_stats ##

接口：The field stats api allows one to find statistical properties of a field without executing a search。比如要查看某索引的timestamp字段情况：则会返回最大值、最小值、以及是否可以聚合统计

GET /nginx-elk-2020.03.24/_field_stats?fields=@timestamp