Logstash 技术实现数据同步快速入门

男娘i 2023-03-04 13:29 6阅读 0赞

# **Logstash 数据同步** #

## 一.**概述** ##

> Logstash是一个开源的服务器端数据处理管道，可以同时从多个数据源获取数据，并对其进行转换，然后将其发送到Elasticsearch中。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNzA1MTMx_size_16_color_FFFFFF_t_70][]

*  Logstash 采用可插拔框架，拥有 200 多个插件。您可以将不同的输入选择、过滤器和输出选择混合搭配、精心安排，让它们在管道中和谐地运行。

## 二.**下载** ##

> https://www.elastic.co/cn/downloads/past-releases\#logstash

![20200803211643502.png][]

*  可自行选择版本下载

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNzA1MTMx_size_16_color_FFFFFF_t_70 1][]

## 三.**安装** ##

**解压即可**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNzA1MTMx_size_16_color_FFFFFF_t_70 2][]

## 四.**数据同步** ##

> Logstash的工作是从MySQL中读取数据，向ES中创建索引，这里需要提前创建mapping的模板文件以便logstash使用。

*  ### 步骤一：拷贝数据库驱动 ###

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNzA1MTMx_size_16_color_FFFFFF_t_70 3][]

*  ### 步骤二：创建  config/xc\_course\_mysql.conf ，用于描述从哪个数据库读取数据 ###
 *  创建的文件必须是UTF-8编码，且无BOM

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNzA1MTMx_size_16_color_FFFFFF_t_70 4][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNzA1MTMx_size_16_color_FFFFFF_t_70 5][]

input {
      stdin {
      }
      jdbc {
    	#serverTimezone=UTC
    	#serverTimezone=Asia/Shanghai
        jdbc_connection_string => "jdbc:mysql://localhost:3306/xc_course?useUnicode=true&characterEncoding=utf-8&useSSL=true&serverTimezone=GMT%2B8"
        # the user we wish to excute our statement as
        jdbc_user => "root"
        jdbc_password => "1234"
        # the path to our downloaded jdbc driver  
        jdbc_driver_library => "../lib/mysql-connector-java-5.1.47.jar"
        # the name of the driver class for mysql
        jdbc_driver_class => "com.mysql.jdbc.Driver"
        #分页设置
        jdbc_paging_enabled => "true"
        jdbc_page_size => "50000"
        jdbc_fetch_size => 100
        #要执行的sql文件
        #statement_filepath => "/conf/course.sql"
        #statement => "select * from course_pub where timestamp > date_add(:sql_last_value,INTERVAL 8 HOUR)"
        statement => "select * from course_pub where timestamp > :sql_last_value"
        #定时配置
        schedule => "* * * * *"
        record_last_run => true
        last_run_metadata_path => "../config/logstash_metadata"
      }
    }
    
    
    output {
      elasticsearch {
        #ES的ip地址和端口
        hosts => "localhost:9200"
        #hosts => ["localhost:9200","localhost:9202","localhost:9203"]
        #ES索引库名称
        index => "xc_course"
        document_id => "%{id}"
        document_type => "CoursePub"
        template =>"../config/xc_course_template.json"
        template_name =>"xc_course"
        template_overwrite =>"true"
      }
      stdout {
       #日志输出
        codec => json_lines
      }
    }

*  ### 步骤三：创建 config/xc\_course\_template.json 用于配置logstash向elasticsearch写入数据的模板 ###

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNzA1MTMx_size_16_color_FFFFFF_t_70 6][]

{
    	"template" : "xc_course",
    	"mappings" : {
    		"CoursePub": {
    			"properties": {
    			  "charge": {
    				"type": "keyword"
    			  },
    			  "description": {
    				"type": "text",
    				"analyzer": "ik_max_word",
    				"search_analyzer": "ik_smart"
    			  },
    			  "end_time": {
    				"type": "date",
    				"format": "yyyy-MM-dd HH:mm:ss"
    			  },
    			  "expires": {
    				"type": "date",
    				"format": "yyyy-MM-dd HH:mm:ss"
    			  },
    			  "grade": {
    				"type": "keyword"
    			  },
    			  "id": {
    				"type": "keyword"
    			  },
    			  "mt": {
    				"type": "keyword"
    			  },
    			  "name": {
    				"type": "text",
    				"analyzer": "ik_max_word",
    				"search_analyzer": "ik_smart"
    			  },
    			  "pic": {
    				"type": "keyword",
    				"index": false
    			  },
    			  "price": {
    				"type": "float"
    			  },
    			  "price_old": {
    				"type": "float"
    			  },
    			  "pub_time": {
    				"type": "date",
    				"format": "yyyy-MM-dd HH:mm:ss"
    			  },
    			  "qq": {
    				"type": "keyword",
    				"index": false
    			  },
    			  "st": {
    				"type": "keyword"
    			  },
    			  "start_time": {
    				"type": "date",
    				"format": "yyyy-MM-dd HH:mm:ss"
    			  },
    			  "status": {
    				"type": "keyword"
    			  },
    			  "studymodel": {
    				"type": "keyword"
    			  },
    			  "teachmode": {
    				"type": "keyword"
    			  },
    			  "teachplan": {
    				"type": "text",
    				"analyzer": "ik_max_word",
    				"search_analyzer": "ik_smart"
    			  },
    			  "users": {
    				"type": "text",
    				"index": false
    			  },
    			  "valid": {
    				"type": "keyword"
    			  }
    			}
    		}
    	}
    }

*  ### 步骤四：启动logstash ###

> logstash.bat -f ..\\config\\xc\_course\_mysql.conf

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNzA1MTMx_size_16_color_FFFFFF_t_70 7][]

*  启动成功:

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNzA1MTMx_size_16_color_FFFFFF_t_70 8][]

*  访问:

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNzA1MTMx_size_16_color_FFFFFF_t_70 9][]

看完恭喜你,又知道了一点点!!!

**你知道的越多,不知道的越多! **

**~感谢志同道合的你阅读,  你的支持是我学习的最大动力 ! 加油 ,陌生人一起努力,共勉!!**

> **注: 如果本篇有需要改进的地方或错误,欢迎大神们指定一二~~**

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNzA1MTMx_size_16_color_FFFFFF_t_70]: /images/20230208/3be5b84d45ca490d8e48ed1a65cfeb88.png
[20200803211643502.png]: /images/20230208/1d8b3b41d82543d89aabe2b8d9086b76.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNzA1MTMx_size_16_color_FFFFFF_t_70 1]: /images/20230208/2745c11d982c46ca98862ed3eb29afce.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNzA1MTMx_size_16_color_FFFFFF_t_70 2]: /images/20230208/ca659635a0184c1e8d1d5495592370b6.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNzA1MTMx_size_16_color_FFFFFF_t_70 3]: /images/20230208/ae6068142acf472ca7211c9f7a1efefb.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNzA1MTMx_size_16_color_FFFFFF_t_70 4]: /images/20230208/5f508ce5d7ac42bc922f40bfaefc8139.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNzA1MTMx_size_16_color_FFFFFF_t_70 5]: /images/20230208/0fd1f4af61c74923b311c3a520865512.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNzA1MTMx_size_16_color_FFFFFF_t_70 6]: /images/20230208/a6720118eaf349809b611cbe2d12bb02.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNzA1MTMx_size_16_color_FFFFFF_t_70 7]: /images/20230208/bc3c89ac075a4a7ab7fce6786503f94a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNzA1MTMx_size_16_color_FFFFFF_t_70 8]: /images/20230208/5dc146bf1ddb47a49cc84fffe45d6fb0.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNzA1MTMx_size_16_color_FFFFFF_t_70 9]: /images/20230208/4484c3d8868140c8bc8fe618174c135a.png