win10下docker操作filebeat做日志采集

我就是我 2023-03-13 15:17 75阅读 0赞

**背景：在[Logstash数据源为日志文件操作][Logstash] 基础上进行**

### 一、 拉取镜像 ###

docker pull elastic/filebeat:7.6.2

### 二、编写filebeat配置文件 ###

**2.1 在目录D:\\usr\\local\\etc\\filebeat下分别创建docker-compose-filebeat.yml、filebeat.yml文件**

**2.2 docker-compose-filebeat.yml文件内容**

version: '3'
    services:
      filebeat:
        image: elastic/filebeat:7.6.2
        container_name: filebeat
        volumes:
            - /d/usr/local/logs/filebeat:/usr/share/filebeat/logs
            - /d/usr/local/etc/filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml
            - /d/usr/local/var/filebeat/data:/usr/share/filebeat/data

**2.3** **filebeat.yml文件内容**

filebeat.inputs:
    - type: log
      enabled: true
      paths:
        - /usr/share/filebeat/logs/*.log
    output.logstash:
      hosts: ["192.168.1.110:5044"]

** 2.4 修改D:\\usr\\local\\etc\\logstash\\pipeline目录下logstash配置文件logstash.conf，新增beats数据源**

input { 
        stdin { } 
        beats {
            port => 5044
            type => "filebeat"
        }
        tcp {
            mode => "server"
            host => "0.0.0.0"
            # 从5044端口取日志
            port => 5044
            # 需要安装logstash-codec-json_lines插件
            codec => json_lines
            type => "tcp"
        }
        file {
            # 容器中日志所在目录的文件
            path => ["/usr/share/logstash/logs/*.log"]
            #codec => "json"
            sincedb_path => "NUL"
            type => "spring"
            start_position => "beginning"
        }
    }
    
    output {
        if [type] == "spring" {
            elasticsearch {
                hosts => ["192.168.1.110:9200"]
                index => "spring-%{+YYYY.MM.dd}"
            }
        }
        if [type] == "tcp" {
            elasticsearch { 
                hosts => ["192.168.1.110:9200"] 
                index => "logstash-%{+YYYY.MM.dd}"  
            }
        }
        if [type] == "filebeat" {
            elasticsearch { 
                hosts => ["192.168.1.110:9200"] 
                index => "%{[@metadata][beat]}-%{[@metadata][version]}"
            }
        }
    	stdout { codec => rubydebug }
    }

### 三、编写脚本 ###

**3.1 编写日志脚本****save\_filebeat\_log.py和启动脚本docker-filebeat-up.sh**

**3.2 save\_filebeat\_log.py文件内容**

#!/usr/bin/python
    # -*- coding: UTF-8 -*-
    import datetime
    import threading
    
    import os
    
    # 递归级联创建的目录
    path = "D:/usr/local/logs/filebeat"
    if not os.path.exists(path):
        os.makedirs(path)
    today = datetime.date.today()
    # 方法1： 此方法文件夹不存在会报错：FileNotFoundError: [Errno 2] No such file or directory
    with open(f'D:/usr/local/logs/filebeat/filebeat-{today}.log', mode='a+', encoding='utf-8') as f:
        now = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f')[:-3]
        thread_id = threading.currentThread().native_id
        log_str = f'{now} DEBUG {thread_id} --- [scheduling-1] org.jooq.tools.LoggerListener            : ' \
                  'Executing query          : select id from user'
        f.writelines(f'{log_str}\n')
        f.close()
        print('保存日志成功')

**3.3 docker-filebeat-up.sh文件内容**

#!/bin/bash
    
    docker-compose -f D:/usr/local/etc/filebeat/docker-compose-filebeat.yml up -d

### 四、启动容器 ###

**4.1 重启logstash容器**

docker restart logstash

![20200515141101862.png][]

**4.2 执行docker-filebeat-up.sh脚本，启动filebeat容器**

./docker-filebeat-up.sh

![20200515141143876.png][]

**4.3 执行save\_filebeat\_log.py脚本，模拟生成日志**

python3 save_filebeat_log.py

![20200515141555865.png][]

### 五、查看结果 ###

**5.1 打开浏览器，查看请求**[http://localhost:9100/][http_localhost_9100]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pzeDE4MjczMTE3MDAz_size_16_color_FFFFFF_t_70][]

**从结果可以看出filebeat日志采集成功 **

[Logstash]: https://blog.csdn.net/zsx18273117003/article/details/106116428
[20200515141101862.png]: /images/20230312/4dae7ffad7e043f08e78320f8f7404d7.png
[20200515141143876.png]: /images/20230312/18dea46b87c94678b92ebf977fd47d24.png
[20200515141555865.png]: /images/20230312/438377fb06a94bb18bf1ce0c93fdfc2f.png
[http_localhost_9100]: http://localhost:9100/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pzeDE4MjczMTE3MDAz_size_16_color_FFFFFF_t_70]: /images/20230312/bff13df3ef4d4b498dfb79d899d0294d.png