Elasticsearch（三）Analysis

青旅半醒 2022-05-10 06:20 245阅读 0赞

### 1.standard analyzer 标准分析器 ###

由以下使用分词器和分词过滤器组成

*  [Standard Tokenizer][]    
 *  [Standard Token Filter][]
 *  [Lower Case Token Filter][]
 *  [Stop Token Filter][]

POST _analyze
    {
      "analyzer": "standard",
      "text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
    }

分词结果:

[ the, 2, quick, brown, foxes, jumped, over, the, lazy, dog's, bone ]

自动小写化分词，默认不适用停用词，可配置三个参数

（1）单个词最大长度（max\_token\_length 默认255，超过则按照255切分）

（2）停用词(stopwords   即配置过滤词如the to等  默认为\_none\_)

（3）停用词文件路径(stopwords\_path   默认没配)

修改配置，需新建一个分析器，如下

PUT my_index
    {
      "settings": {
        "analysis": {
          "analyzer": {
            "my_standard_analyzer": {
              "type": "standard",
              "max_token_length": 10,
              "stopwords": "_english_"
            }
          }
        }
      }
    }

通过以上配置，在my\_index中新增了一个my\_standard\_analyzer分析器，之后字段定义可直接使用此分析器。

stopwords支持一些国家的语言进行分词，\_english\_为标准英文单词分词，支持如下，不支持中文。

`_arabic_`, `_armenian_`, `_basque_`, `_bengali_`, `_brazilian_`, `_bulgarian_`, `_catalan_`, `_czech_`, `_danish_`, `_dutch_`, `_english_`, `_finnish_`, `_french_`, `_galician_`, `_german_`, `_greek_`, `_hindi_`, `_hungarian_`, `_indonesian_`, `_irish_`, `_italian_`, `_latvian_`, `_norwegian_`, `_persian_`, `_portuguese_`, `_romanian_`, `_russian_`, `_sorani_`, `_spanish_`, `_swedish_`, `_thai_`, `_turkish_`

还可以通过如下配置该分析器具体过滤的词

"stopwords": ["and", "is", "the"]

使用当前配置分词结果如下

[ quick, brown, foxes, jumped, over, lazy, dog, s, bone ]

### 2.simple analyzer 简单分析器 ###

由以下使用分词器组成

*  [Lower Case Tokenizer][]

只支持自动小写化词，不支持配置。

### 3.whitespace  analyzer  空白分析器 ###

由以下使用分词器组成

*  [Whitespace Tokenizer][]

区分大小写分词，不做小写化处理，分词结果如下

[ The, 2, QUICK, Brown-Foxes, jumped, over, the, lazy, dog's, bone. ]

### 4.stop  analyzer  停用分析器 ###

由以下使用分词器和分词过滤器组成

*  [Lower Case Tokenizer][]
 *  [Stop Token Filter][]

默认自动小写化词，默认使用`_english_` 方式停用词。

可配置，配置参数stopwords和stopwords\_path，与standard 分析器配置方式一致，参考1例子。

### 5.keyword  analyzer 关键字分析器  ###

由以下使用分词器组成

*  [Keyword Tokenizer][]

无自动小写化，返回整段词，无配置，分析结果如下。

[ The 2 QUICK Brown-Foxes jumped over the lazy dog's bone. ]

### 6.pattern analyzer  模式分析器 ###

由以下使用分词器和分词过滤器组成

*  [Pattern Tokenizer][]
 *  [Lower Case Token Filter][]
 *  [Stop Token Filter][]

自动小写化，通过正则匹配分割词，默认是\\W+,即匹配所有非数字、字母和下划线的字符,默认分词结果如下。

[ the, 2, quick, brown, foxes, jumped, over, the, lazy, dog, s, bone ]

可配置参数如下

（1）pattern   匹配分隔符的正则表达式，默认\\W+，正则可参考此链接[Java regular expression][]

（2）flags    匹配模式,多个用|分隔（CASE\_INSENSITIVE|COMMENTS），参考java Pattern类field，[入口][Link 1]

（3）lowercase   自动小写化，默认true

（4）stopwords    停用词

（5）stopwords\_path   停用词文件路径

配置例子如下：以所有非字母和数字作为切割符，自动小写化

PUT my_index
    {
      "settings": {
        "analysis": {
          "analyzer": {
            "my_email_analyzer": {
              "type":      "pattern",
              "pattern":   "\W|_", 
              "lowercase": true
            }
          }
        }
      }
    }

### 7.fingerprint  analyzers ###

由以下使用分词器和分词过滤器组成

*  [Standard Tokenizer][]
 *  [Lower Case Token Filter][]
 *  [ASCII Folding Token Filter][]
 *  [Stop Token Filter][]
 *  [Fingerprint Token Filter][]

返回单一结果，结果自动排序，去除重复词。

POST _analyze
    {
      "analyzer": "fingerprint",
      "text": "Yes yes, Gödel said this sentence is consistent and."
    }

[ and consistent godel is said sentence this yes ]

配置参数max\_output\_size,stopwords,stopwords\_path与标准分析器一致

separator参数    分词结果中每个单词的分隔方式，默认是空格

### 8.custom analyzer 自定义分析器 ###

通过配置[tokenizer][]、[character filters][]、[token filters][]实现定制的分析器。

PUT my_index
    {
      "settings": {
        "analysis": {
          "analyzer": {
            "my_custom_analyzer": {
              "type":      "custom", 
              "tokenizer": "standard",
              "char_filter": [
                "html_strip"
              ],
              "filter": [
                "lowercase",
                "asciifolding"
              ]
            }
          }
        }
      }
    }
    
    POST my_index/_analyze
    {
      "analyzer": "my_custom_analyzer",
      "text": "Is this <b>déjà vu</b>?"
    }

分词结果

[ is, this, deja, vu ]

参考自官方文档: [https://www.elastic.co/guide/en/elasticsearch/reference/6.x/analysis.html][https_www.elastic.co_guide_en_elasticsearch_reference_6.x_analysis.html]

[Standard Tokenizer]: https://www.elastic.co/guide/en/elasticsearch/reference/6.x/analysis-standard-tokenizer.html
[Standard Token Filter]: https://www.elastic.co/guide/en/elasticsearch/reference/6.x/analysis-standard-tokenfilter.html
[Lower Case Token Filter]: https://www.elastic.co/guide/en/elasticsearch/reference/6.x/analysis-lowercase-tokenfilter.html
[Stop Token Filter]: https://www.elastic.co/guide/en/elasticsearch/reference/6.x/analysis-stop-tokenfilter.html
[Lower Case Tokenizer]: https://www.elastic.co/guide/en/elasticsearch/reference/6.x/analysis-lowercase-tokenizer.html
[Whitespace Tokenizer]: https://www.elastic.co/guide/en/elasticsearch/reference/6.x/analysis-whitespace-tokenizer.html
[Keyword Tokenizer]: https://www.elastic.co/guide/en/elasticsearch/reference/6.x/analysis-keyword-tokenizer.html
[Pattern Tokenizer]: https://www.elastic.co/guide/en/elasticsearch/reference/6.x/analysis-pattern-tokenizer.html
[Java regular expression]: http://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html
[Link 1]: https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html#field.summary
[ASCII Folding Token Filter]: https://www.elastic.co/guide/en/elasticsearch/reference/6.x/analysis-asciifolding-tokenfilter.html
[Fingerprint Token Filter]: https://www.elastic.co/guide/en/elasticsearch/reference/6.x/analysis-fingerprint-tokenfilter.html
[tokenizer]: https://www.elastic.co/guide/en/elasticsearch/reference/6.x/analysis-tokenizers.html
[character filters]: https://www.elastic.co/guide/en/elasticsearch/reference/6.x/analysis-charfilters.html
[token filters]: https://www.elastic.co/guide/en/elasticsearch/reference/6.x/analysis-tokenfilters.html
[https_www.elastic.co_guide_en_elasticsearch_reference_6.x_analysis.html]: https://www.elastic.co/guide/en/elasticsearch/reference/6.x/analysis.html