ELK 7.5.0（五）Logstash正则分析提取日志，优化日志记录

秒速五厘米 2023-06-19 07:23 51阅读 0赞

## 一、实验环境 ##

主机名        IP
    es        192.168.14.210
    kibana    192.168.14.210
    logstash  192.168.14.211
    nginx     192.168.14.211

### **优化说明：** ###

删除字段是为了减少es的数据库大小

提取添加字段是为了kibana更加明确的展示，也有助于创建可视化视图提供依据。

## 二、Nginx编译安装 ##

1、安装依赖环境

[root@logstash ~]# yum install -y lrzsz wget gcc gcc-c++ make pcre pcre-devel zlib zlib-devel

2、解压缩包

[root@logstash ~]# tar -zxvf nginx-1.14.2.tar.gz

3、简单编译安装

[root@logstash ~]# cd nginx-1.14.2
    [root@logstash ~]# ./configure --prefix=/usr/local/nginx && make && make install

4、设置环境变量并启动

[root@logstash ~]# vi /etc/profile
    export PATH=$PATH:/usr/local/nginx/sbin/
    [root@logstash ~]# source  /etc/profile 
    
    #检测配置文件正常就启动
    [root@logstash ~]# nginx -t
    nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
    nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
    [root@logstash ~]# nginx

5、Nginx日志格式说明

192.168.5.53 - - [05/Dec/2019:08:17:11 +0800] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" "-"
    192.168.5.53 - - [05/Dec/2019:08:17:11 +0800] "GET /favicon.ico HTTP/1.1" 404 571 "http://192.168.14.211/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" "-"
    
    从左往右分别是：
    访问IP地址
    访问时间
    请求方式(GET/POST)
    请求URL
    状态码
    响应body大小
    Referer
    User Agent

## 三、Logstash正则提取日志 ##

1、提取客户端IP

参考kibana的Grok Debugger了解语法：

Grok使用(?<xxx>提取内容)来提取xxx字段

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70][]

2、grok提取nginx日志（tomcat等日志使用方法类似）

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 1][]

3、修改Logstash配置文件

[root@logstash ~]# vi /usr/local/logstash-7.5.0/config/logstash.conf 
    input {
      file {
        path => "/usr/local/nginx/logs/access.log"        #nginx日志路径
      }
    }
    filter {
        grok {
            match => {
                "message" => '(?<clientip>[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) - - \[(?<requesttime>[^ ]+ \+[0-9]+)\] "(?<requesttype>[A-Z]+) (?<requesturl>[^ ]+) HTTP/\d.\d" (?<status>[0-9]+) (?<bodysize>[0-9]+) "[^"]+" "(?<ua>[^"]+)"'
            } 
        }
    }
    output {
      elasticsearch {
        hosts => ["http://192.168.14.210:9200"]            #es服务器IP地址
        user => "elastic"
        password => "elkpwd"
        index => "logstash-%{+YYYY.MM.dd}"                 #索引格式
      }
    }

4、Logstash后台启动

[root@logstash ~]# nohup logstash -f /usr/local/logstash-7.5.0/config/logstash.conf  >/tmp/logstash.log 2>/tmp/logstash.log &
    
    
    #如果已经启动，可以直接重载配置
    [root@logstash ~]# kill -1 11863   #11863是Logstash进程ID

## 四、kibana查看索引的字段 ##

1、为了区别明显，先删除旧的日志索引

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 2][]

2、访问nginx使其产生日志，添加新的日志索引（如果日志索引只有名称没有日期，可以停止Logstash再启动）

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 3][]

3、根据自动搜索的索引格式，可以是logstash\*或者是logstash-日期区分

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 4][]

4、选择时间轴

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 5][]

5、查看到logstash自定义的字段

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 6][]

## 五、Logstash删除无需记录字段（只能去除\_source里的） ##

1、删除之前的字段

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 7][]

2、修改logstash配置文件，删除无需记录字段

[root@logstash ~]# vi  /usr/local/logstash-7.5.0/config/logstash.conf 
    remove_field => ["message","@version","path"]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 8][]

3、重载logstash配置，如不行就停止重新启动

[root@logstash ~]# ps auxf|grep logstash 
    root     12830 14.0 10.1 4909284 811844 pts/0  Sl   15:09   3:12 ...忽略
    
    [root@logstash ~]# kill -1 12830        #进程ID

4、删除之后的字段

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 9][]

至此完成

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20191205122839519.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20191205123345657.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20191205150440985.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/2019120513460327.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20191205151121679.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20191205151301937.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/20191205151605132.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 7]: https://img-blog.csdnimg.cn/20191205153606329.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 8]: https://img-blog.csdnimg.cn/2019120515310588.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 9]: https://img-blog.csdnimg.cn/20191205154357760.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv,size_16,color_FFFFFF,t_70