hive数据通过hfile导入hbase

太过爱你忘了你带给我的痛 2023-02-28 05:52 51阅读 0赞

# hive数据通过hfile导入hbase #

## **1、原始数据表** ##

CREATE TABLE `student`(
    `name` string,
    `birthday` string,
    `localtion` string
    )

## **2、创建hive表** ##

CREATE TABLE `student_info`(
    `rowkey` string,
    `name` string,
    `birthday` string,
    `localtion` string
    )
    STORED AS
    INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat'
    OUTPUTFORMAT 'org.apache.hadoop.hive.hbase.HiveHFileOutputFormat'
    TBLPROPERTIES ('hfile.family.path' = '/tmp/student/baseInfo');

## **3、添加jar包** ##

*  需要将hive/lib下的所有关于hbase的jar

ll lib | grep hbase

![\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-D4SLD1Ar-1595235719138)(C:\\Users\\wandu\\AppData\\Roaming\\Typora\\typora-user-images\\1595056887181.png)\]][img-D4SLD1Ar-1595235719138_C_Users_wandu_AppData_Roaming_Typora_typora-user-images_1595056887181.png]

*  在hive中

add jar /home/hadoop/work/apps/hive-3.1.2/lib/hbase-client-2.0.0-alpha4.jar ;
    add jar /home/hadoop/work/apps/hive-3.1.2/lib/hbase-common-2.0.0-alpha4.jar ;
    add jar /home/hadoop/work/apps/hive-3.1.2/lib/hbase-common-2.0.0-alpha4-tests.jar ;
    add jar /home/hadoop/work/apps/hive-3.1.2/lib/hbase-hadoop2-compat-2.0.0-alpha4.jar ;
    add jar /home/hadoop/work/apps/hive-3.1.2/lib/hbase-hadoop2-compat-2.0.0-alpha4-tests.jar;
    add jar /home/hadoop/work/apps/hive-3.1.2/lib/hbase-hadoop-compat-2.0.0-alpha4.jar ;
    add jar /home/hadoop/work/apps/hive-3.1.2/lib/hbase-http-2.0.0-alpha4.jar  ;
    add jar /home/hadoop/work/apps/hive-3.1.2/lib/hbase-mapreduce-2.0.0-alpha4.jar ;
    add jar /home/hadoop/work/apps/hive-3.1.2/lib/hbase-metrics-2.0.0-alpha4.jar ;
    add jar /home/hadoop/work/apps/hive-3.1.2/lib/hbase-metrics-api-2.0.0-alpha4.jar ;
    add jar /home/hadoop/work/apps/hive-3.1.2/lib/hbase-prefix-tree-2.0.0-alpha4.jar ;
    add jar /home/hadoop/work/apps/hive-3.1.2/lib/hbase-procedure-2.0.0-alpha4.jar ;
    add jar /home/hadoop/work/apps/hive-3.1.2/lib/hbase-protocol-2.0.0-alpha4.jar ;
    add jar /home/hadoop/work/apps/hive-3.1.2/lib/hbase-protocol-shaded-2.0.0-alpha4.jar ;
    add jar /home/hadoop/work/apps/hive-3.1.2/lib/hbase-replication-2.0.0-alpha4.jar ;
    add jar /home/hadoop/work/apps/hive-3.1.2/lib/hbase-server-2.0.0-alpha4.jar ;
    add jar /home/hadoop/work/apps/hive-3.1.2/lib/hbase-shaded-miscellaneous-1.0.1.jar ;
    add jar /home/hadoop/work/apps/hive-3.1.2/lib/hbase-shaded-netty-1.0.1.jar ;
    add jar /home/hadoop/work/apps/hive-3.1.2/lib/hbase-shaded-protobuf-1.0.1.jar ;
    add jar /home/hadoop/work/apps/hive-3.1.2/lib/hive-hbase-handler-3.1.2.jar ;

## 4、导入数据 ##

insert into table student_info 
    select 
    concat(name, '_', birthday, '_', localtion) as rowkey,
    name,
    birthday,
    localtion
    from student
    order by rowkey
    ;

## 5、查看hdfs数据 ##

hdfs dfs -ls /tmp/student/baseInfo

## 6、通过bulkload导入到hbase ##

创建对应的hbase表，其中baseInfo为family，应和上一步对应

create 'student_info','baseInfo'

执行bulkhead，将数据导入到hbase中

bin/hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles hdfs://master:9000/tmp/student  student_info

注意hdfs目录的baseInfo为hbase的列族，bulkhead是不加

## 7、hbase查看数据 ##

scan 'student_info'

注意hdfs目录的baseInfo为hbase的列族，bulkhead是不加
    
    
    
    ## 7、hbase查看数据
    
    ```shell
    scan 'student_info'

[img-D4SLD1Ar-1595235719138_C_Users_wandu_AppData_Roaming_Typora_typora-user-images_1595056887181.png]: /images/20230209/efb57e747443472b9812d8439453faae.png