Hive导入数据到Elasticsearch

痛定思痛。 2023-02-22 15:56 63阅读 0赞

本地环境试验大家可以用docker快速搭建hive服务和es服务，可参照我的上两篇博客。

登录到hive server宿主机或者容器。

有网络直接下载相关对应版本es hadoop jar包。

[https://repo1.maven.org/maven2/org/elasticsearch/elasticsearch-hadoop/7.6.2/elasticsearch-hadoop-7.6.2.jar][https_repo1.maven.org_maven2_org_elasticsearch_elasticsearch-hadoop_7.6.2_elasticsearch-hadoop-7.6.2.jar]

$ winpty docker exec -it 1f588dced43d bash
    root@1f588dced43d:/opt# ls
    elasticsearch-hadoop-7.6.2.jar  hadoop-2.7.4  hive
    root@1f588dced43d:/opt# hive
    SLF4J: Class path contains multiple SLF4J bindings.
    SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/opt/hadoop-2.7.4/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBin
    der.class]
    SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
    SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
    
    Logging initialized using configuration in file:/opt/hive/conf/hive-log4j2.properties Async: true
    Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e
    . spark, tez) or using Hive 1.X releases.
    hive> add jar /opt/elasticsearch-hadoop-7.6.2.jar
        > ;
    Added [/opt/elasticsearch-hadoop-7.6.2.jar] to class path
    Added resources: [/opt/elasticsearch-hadoop-7.6.2.jar]
    hive> create table test(uid string,uname string,udate string) STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' TBLPROPERTI
    ES('es.resource' = 'test/user', 'es.nodes'='192.168.1.4', 'es.port'='9200', 'es.nodes.wan.only'='true');
    OK
    Time taken: 0.659 seconds
    hive> show tables;
    OK
    test
    user
    Time taken: 0.071 seconds, Fetched: 2 row(s)
    hive> insert into table test select * from `user`;
    WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution en
    gine (i.e. spark, tez) or using Hive 1.X releases.
    Query ID = root_20200704083011_352e0344-ea3c-487b-b97c-0b9f1ecf070f
    Total jobs = 1
    Launching Job 1 out of 1
    Number of reduce tasks is set to 0 since there's no reduce operator
    Job running in-process (local Hadoop)
    2020-07-04 08:30:14,167 Stage-3 map = 100%,  reduce = 0%
    Ended Job = job_local1424993503_0001
    MapReduce Jobs Launched:
    Stage-Stage-3:  HDFS Read: 150 HDFS Write: 0 SUCCESS
    Total MapReduce CPU Time Spent: 0 msec
    OK
    Time taken: 2.759 seconds
    hive>

[https_repo1.maven.org_maven2_org_elasticsearch_elasticsearch-hadoop_7.6.2_elasticsearch-hadoop-7.6.2.jar]: https://repo1.maven.org/maven2/org/elasticsearch/elasticsearch-hadoop/7.6.2/elasticsearch-hadoop-7.6.2.jar