ELK流程图以及部分搭建

本是古典何须时尚 2023-07-09 09:28 55阅读 0赞

### 大致框架架构 ###

ELK是一组开源软件的简称，其包括Elasticsearch、Logstash 和 Kibana。ELK最近几年发展迅速，已经成为目前最流行的集中式日志解决方案。

  Elasticsearch: 能对大容量的数据进行接近实时的存储，搜索和分析操作。 本项目中主要通过Elasticsearch存储所有获取的日志。

  Logstash： 数据收集引擎，它支持动态的的从各种数据源获取数据，并对数据进行过滤，分析，丰富，统一格式等操作，然后存储到用户指定的位置。

  Kibana： 数据分析与可视化平台，对Elasticsearch存储的数据进行可视化分析，通过表格的形式展现出来。

  Filebeat: 轻量级的开源日志文件数据搜集器。通常在需要采集数据的客户端安装Filebeat,并指定目录与日志格式,Filebeat就能快速收集数据，并发送给logstash进行解析，或是直接发给Elasticsearch存储。

  Redis：NoSQL数据库(key-value)，也数据轻型消息队列，不仅可以对高并发日志进行削峰还可以对整个架构进行解耦

流程图：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYW5nc2huZw_size_16_color_FFFFFF_t_70][]

目前的架构流程如下：

Filebeat 已经完全替代了 Logstash 成为新一代的日志采集器，同时鉴于它轻量、安全等特点，经消息队列输出插件输出到消息队列中。目前 Logstash 支持Redis等常见消息队列。然后 Logstash 通过消息队列输入插件从队列中获取数据，分析过滤后经输出插件发送到 Elasticsearch，最后通过 Kibana 展示

## Filebeat设置 ##

修改配置文件filebeat.yml：

\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\# Filebeat Configuration Example \#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#

\# This file is an example configuration file highlighting only the most common  
\# options. The filebeat.reference.yml file from the same directory contains all the  
\# supported options with more comments. You can use it as a reference.  
\#  
\# You can find the full configuration reference here:  
\# https://www.elastic.co/guide/en/beats/filebeat/index.html

\# For more available modules and options, please see the filebeat.reference.yml sample  
\# configuration file.

\#=========================== Filebeat inputs =============================

filebeat.inputs:

\# Each - is an input. Most options can be set at the input level, so  
\# you can use different inputs for various configurations.  
\# Below are the input specific configurations.

\- type: log  
  enabled: true  
  paths:  
    - /opt/data/cjwl56log/log/\*/log-info.\*  
    - /opt/data/cjwl56log/log/\*/log-error.\*  
  fields:  
    log\_source: action  
    log\_type: action  
  tail\_files: true  
  scan\_frequency: 60s  
  backoff: 10s

\- type: log  
  enabled: true  
  paths:  
    - /opt/data/logs/api-gateway/\*/log-info.\*  
    - /opt/data/logs/api-gateway/\*/log-error.\*  
  fields:  
    log\_source: gateway  
    log\_type: gateway  
  tail\_files: true  
  scan\_frequency: 60s  
  backoff: 10s

\- type: log  
  enabled: true  
  paths:  
    - /opt/data/logs/cjkj-oauth-service/\*/log-info.\*  
    - /opt/data/logs/cjkj-oauth-service/\*/log-error.\*  
  fields:  
    log\_source: oauth  
    log\_type: oauth  
  tail\_files: true  
  scan\_frequency: 60s  
  backoff: 10s

\- type: log  
  enabled: true  
  paths:  
    - /opt/data/logs/cjkj-usercenter-service/\*/log-info.\*  
    - /opt/data/logs/cjkj-usercenter-service/\*/log-error.\*  
  fields:  
    log\_source: usercenter  
    log\_type: usercenter  
  tail\_files: true  
  scan\_frequency: 60s  
  backoff: 10s

\# Exclude lines. A list of regular expressions to match. It drops the lines that are  
  \# matching any regular expression from the list.  
  \#exclude\_lines: \['^DBG'\]

\# Include lines. A list of regular expressions to match. It exports the lines that are  
  \# matching any regular expression from the list.  
  \#include\_lines: \['^ERR', '^WARN'\]

\# Exclude files. A list of regular expressions to match. Filebeat drops the files that  
  \# are matching any regular expression from the list. By default, no files are dropped.  
  \#exclude\_files: \['.gz$'\]

\# Optional additional fields. These fields can be freely picked  
  \# to add additional information to the crawled log files for filtering  
  \#fields:  
  \#  level: debug  
  \#  review: 1

\#\#\# Multiline options

\# Multiline can be used for log messages spanning multiple lines. This is common  
  \# for Java Stack Traces or C-Line Continuation

\# The regexp Pattern that has to be matched. The example pattern matches all lines starting with \[  
  \#multiline.pattern: ^\\\[

\#错误日志合并   或者说多行日志合并

\# Defines if the pattern set under pattern should be negated or not. Default is false.  
  \#multiline.negate: false  
  multiline.negate: true  
  multiline.pattern: '^\[0-9\]\{2\}:\[0-9\]\{2\}:\[0-9\]\{2\}'  
  multiline.match: after  
  multiline.max\_lines: 100  
  \# Match can be set to "after" or "before". It is used to define if lines should be append to a pattern  
  \# that was (not) matched before or after or as long as a pattern is not matched based on negate.  
  \# Note: After is the equivalent to previous and before is the equivalent to to next in Logstash  
  \#multiline.match: after

\#============================= Filebeat modules ===============================

filebeat.config.modules:  
  \# Glob pattern for configuration loading  
  path: $\{path.config\}/modules.d/\*.yml

\# Set to true to enable config reloading  
  reload.enabled: false

\# Period on which files under path should be checked for changes  
  \#reload.period: 10s

\#==================== Elasticsearch template setting ==========================

setup.template.settings:  
  index.number\_of\_shards: 1  
  \#index.codec: best\_compression  
  \#\_source.enabled: false

\#================================ General =====================================

\# The name of the shipper that publishes the network data. It can be used to group  
\# all the transactions sent by a single shipper in the web interface.  
\#name:

\# The tags of the shipper are included in their own field with each  
\# transaction published.  
\#tags: \["service-X", "web-tier"\]

\# Optional fields that you can specify to add additional information to the  
\# output.  
\#fields:  
\#  env: staging

\#============================== Dashboards =====================================  
\# These settings control loading the sample dashboards to the Kibana index. Loading  
\# the dashboards is disabled by default and can be enabled either by setting the  
\# options here or by using the \`setup\` command.  
\#setup.dashboards.enabled: false

\# The URL from where to download the dashboards archive. By default this URL  
\# has a value which is computed based on the Beat name and version. For released  
\# versions, this URL points to the dashboard archive on the artifacts.elastic.co  
\# website.  
\#setup.dashboards.url:

\#============================== Kibana =====================================

\# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.  
\# This requires a Kibana endpoint configuration.  
setup.kibana:

\# Kibana Host  
  \# Scheme and port can be left out and will be set to the default (http and 5601)  
  \# In case you specify and additional path, the scheme is required: http://localhost:5601/path  
  \# IPv6 addresses should always be defined as: https://\[2001:db8::1\]:5601  
  \#host: "localhost:5601"

\# Kibana Space ID  
  \# ID of the Kibana Space into which the dashboards should be loaded. By default,  
  \# the Default Space will be used.  
  \#space.id:

\#============================= Elastic Cloud ==================================

\# These settings simplify using filebeat with the Elastic Cloud (https://cloud.elastic.co/).

\# The cloud.id setting overwrites the \`output.elasticsearch.hosts\` and  
\# \`setup.kibana.host\` options.  
\# You can find the \`cloud.id\` in the Elastic Cloud web UI.  
\#cloud.id:

\# The cloud.auth setting overwrites the \`output.elasticsearch.username\` and  
\# \`output.elasticsearch.password\` settings. The format is \`<user>:<pass>\`.  
\#cloud.auth:

\#================================ Outputs =====================================

\# Configure what output to use when sending the data collected by the beat.

\#-------------------------- Elasticsearch output ------------------------------  
\#output.elasticsearch:  
  \# Array of hosts to connect to.  
\#  hosts: \["localhost:9200"\]

\# Optional protocol and basic auth credentials.  
  \#protocol: "https"  
  \#username: "elastic"  
  \#password: "changeme"

\#----------------------------- Logstash output --------------------------------  
\#output.logstash:  
  \# The Logstash hosts  
  \#hosts: \["10.253.96.81:5044"\]  
  \#index: "dev"     \#索引根名称  
  \# Optional SSL. By default is off.  
  \# List of root certificates for HTTPS server verifications  
  \#ssl.certificate\_authorities: \["/etc/pki/root/ca.pem"\]

\# Certificate for SSL client authentication  
  \#ssl.certificate: "/etc/pki/client/cert.pem"

\# Client Certificate Key  
  \#ssl.key: "/etc/pki/client/cert.key"

\#-------------------------------   redis output ------------------------------  
\#output.redis:  
  \#hosts: \[""\] \#输出到redis的机器  
  \#password: Qaz123$%^Qaz123$%^Qaz123$%^Qaz123$%^Qaz123$%^  
  \#db: 1  \#redis数据库的一个整数索引标识，redis总共0-15默认16个库。  
  \#timeout: 5  
  \#key: "default\_list"  \#以default\_list的keys传输到redis  
  \#reconnect\_interval: 1

\#-------------------------------   kafka  output------------------------------  
output.kafka:  
  enabled: true  
  max\_retries: 5  
  hosts: \["x.x.x.x:9092","x.x.x.x::9092","1x.x.x.x::9092"\]   
  timeout: 300  
  topic: "prod\_filebeatlog"

\#================================ Processors =====================================

\# Configure processors to enhance or manipulate events generated by the beat.

processors:  
  - add\_host\_metadata: ~  
  - add\_cloud\_metadata: ~

\#================================ Logging =====================================

\# Sets log level. The default log level is info.  
\# Available log levels are: error, warning, info, debug  
\#logging.level: debug

\# At debug level, you can selectively enable logging only for some components.  
\# To enable all selectors use \["\*"\]. Examples of other selectors are "beat",  
\# "publish", "service".  
\#logging.selectors: \["\*"\]

\#============================== Xpack Monitoring ===============================  
\# filebeat can export internal metrics to a central Elasticsearch monitoring  
\# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The  
\# reporting is disabled by default.

\# Set to true to enable the monitoring reporter.  
\#monitoring.enabled: false

\# Uncomment to send the metrics to Elasticsearch. Most settings from the  
\# Elasticsearch output are accepted here as well.  
\# Note that the settings should point to your Elasticsearch \*monitoring\* cluster.  
\# Any setting that is not set is automatically inherited from the Elasticsearch  
\# output configuration, so if you have the Elasticsearch output configured such  
\# that it is pointing to your Elasticsearch monitoring cluster, you can simply  
\# uncomment the following line.  
\#monitoring.elasticsearch:

\#================================= Migration ==================================

\# This allows to enable 6.7 migration aliases  
\#migration.6\_to\_7.enabled: true

注释掉Elasticsearch output：下的设置

其他的默认即可。

## Logstash设置 ##

input \{

kafka \{

bootstrap\_servers => "1x.x.x.x:9092,x.x.x.x:9092,x.x.x.x:9092"

topics => \["filebeatlog"\]

\#group\_id => “test-consumer-group”

codec => "json"

consumer\_threads => 3

decorate\_events => true

\}\}

设置索引：

output \{

\# log\_type和之前配置的自定义字段对应

if \[fields\]\[log\_type\] == "action" \{

elasticsearch \{

hosts => \["http://x.x.x.x:9200","http://x.x.x.x.123:9200","http://x.x.x.x:9200"\]

index => "action\_test05" \#定义一个索引的名称

if \[fields\]\[log\_type\] == "gateway" \{

elasticsearch \{

hosts => \["http://x.x.x.x:9200","http://x.x.x.x:9200","http://x.x.x.x:9200"\]

index => "gateway\_test5"

## Kibana设置 ##

server.port: 5601

\#开启默认端口5601

server.host: “x.x.x.x”                       \#kibana站点IP

elasticsearch.url: [http://x.x.x.x:9200][http_x.x.x.x_9200]        \#只想ES服务所在IP Port

kibana.index: “.kibana”

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYW5nc2huZw_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200228154725630.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYW5nc2huZw==,size_16,color_FFFFFF,t_70
[http_x.x.x.x_9200]: http://x.x.x.x:9200/