2.ES----Logstash 安装与测试数据导入

谁践踏了优雅 2023-06-23 12:59 38阅读 0赞

1.下载Logstash 安装包注意版本对应,解压

2.下载movieton测试数据 ,解压得到 movies.csv 文件,复制到logstash的bin目录下

3.在config目录下 新建文件 logstash.conf文件

# Sample Logstash configuration for creating a simple
    # Beats -> Logstash -> Elasticsearch pipeline.
    
    input {
      file {
      #这里指定需要导入的文件的路径 lunix 从/xx/xx
        path => "D:/logstash-7.5.0/logstash-7.5.0/bin/movies.csv"
        start_position => "beginning"
       # 监听文件读取信息记录的位置 win
       #sincedb_path => "H:/ElasticSearch/6.4.3/logstash-6.4.3/test/since_db.txt"
       #Linux
       #sincedb_path => "/dev/null"
      }
    }
    filter {
      csv {
        separator => ","
        columns => ["id","content","genre"]
      }
    
      mutate {
        split => { "genre" => "|" }
        remove_field => ["path", "host","@timestamp","message"]
      }
    
      mutate {
    
        split => ["content", "("]
        add_field => { "title" => "%{[content][0]}"}
        add_field => { "year" => "%{[content][1]}"}
      }
    
      mutate {
        convert => {
          "year" => "integer"
        }
        strip => ["title"]
        remove_field => ["path", "host","@timestamp","message","content"]
      }
    
    }
    output {
       elasticsearch {
       #es地址
         hosts => "http://localhost:9200"
         # 导入文件索引
         index => "movies"
         #数据id
         document_id => "%{id}"
       }
      stdout {}
    }

启动 es ,kibana

以logstash.conf作为配置文件启动 logstash,  
在logstash的bin目录下执行

logstash.bat -f ../config/logstash.conf

如果未执行成功,第二次导入时一定要先把  
sincedb\_path指定的文件删除,否则还是导入不了  
导入成功,cmd控制台会打印记录  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L20wXzM3OTcwNjk5_size_16_color_FFFFFF_t_70]

导入成功后kibanna中  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L20wXzM3OTcwNjk5_size_16_color_FFFFFF_t_70 1]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L20wXzM3OTcwNjk5_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20191219104135907.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L20wXzM3OTcwNjk5,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L20wXzM3OTcwNjk5_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20191219103953654.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L20wXzM3OTcwNjk5,size_16,color_FFFFFF,t_70