hbase错误排查/数据恢复

古城微笑少年丶 2022-03-11 15:52 1598阅读 0赞

### 文章目录 ###

*  1，hbase报错集
 *   *  hbase数据和元数据不吻合，常见错误
     *  解决hbase报错：(同版本的CDH数据转移,只需转移/hbase/data/default, 然后删除并重建本地/hbaes目录)
 *  2，hbase表：数据结构
 *   *  1， 如何查看hbase表的region
     *  2, hfile小文件太多，会影响io读写效率，需要合并
     *  3, region太大，需要手动切分

# 1，hbase报错集 #

*  memstore频繁flush，同时多次compact小文件hfile,导致regionserver宕机  
    解决： 调大hbase.hregion.memstore.flush.size :128M–> 256M

2019-09-09 10:45:31,548 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: yhemr-app01,60020,1565686069646-MemstoreFlusherChore requesting flush of 某hbase表名,6000,1558431459090.62d7cc551574205993ce14d8528383ef. because f has an old edit so flush to free WALs after random delay 64839ms
    2019-09-09 10:45:42,006 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: yhemr-app01,60020,1565686069646-MemstoreFlusherChore requesting flush of 某hbase表名,6000,1558431459090.62d7cc551574205993ce14d8528383ef. because f has an old edit so flush to free WALs after random delay 258291ms
    .......
    2019-09-09 10:47:01,878 INFO org.apache.hadoop.hbase.regionserver.HRegion: Flushing 1/1 column families, memstore=7.86 MB
    2019-09-09 10:50:34,661 INFO org.apache.hadoop.hbase.regionserver.HRegion: Flushing 1/1 column families, memstore=27.08 MB

*  failed transitioning from OPENING to FAILED\_OPEN in ZK, expecting version 28

Opening of region {ENCODED => 76db1967168b933ce73a48f62f352fdb, NAME => 'CDR_ADMISSION_INFO,6000,1547555336020.76db1967168b933ce73a48f62f352fdb.', STARTKEY => '6000', ENDKEY => '9000'} failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version 28

## hbase数据和元数据不吻合，常见错误 ##

由于region发生split, regionserver暂忙，停止了服务，而数据还在往旧的region中插入，导致如下各种错误

*  hfile newer than current time

2019-03-07 22:29:57,128 WARN org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner: Found a hfile (hdfs://nameservice1/hbase/archive/data/hbase/meta/1588230740/info/4eb9725dc9394cfdac00d939a017b83e) newer than current time (1551968997128 < 1552035146004), probably a clock skew
    2019-03-07 22:29:57,128 WARN org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner: Found a hfile (hdfs://nameservice1/hbase/archive/data/hbase/meta/1588230740/info/f89aa07ca82f4f3d924d538897bc622e) newer than current time (1551968997128 < 1552035146001), probably a clock skew

*  NotServingRegionException: Region is not online

ERROR: org.apache.hadoop.hbase.NotServingRegionException: Region t1,3000,1547555336020.1f429ac99ee4f23c509a6672edcd4ea2. is not online on yhemr-app01,60020,1551956278506
    ERROR: org.apache.hadoop.hbase.NotServingRegionException: Region t2,9000,1548143206562.b5cb9d9221a5df0c9e4efcb9a947cc9a. is not online on yhemr-app2,60020,1551956278838

*  hbase:namespace region SocketTimeoutException

java.net.SocketTimeoutException: callTimeout=1200000, callDuration=1215227: row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1544509340812.9bba7be99574bf5dae25b895c08e6d24., hostname=yhemr-app2,60020,1552034802552, seqNum=52

*  Failed open of region,starting to roll back the global memstore size

Failed open of region=CDR_ADMISSION_INFO,6000,1547555336020.76db1967168b933ce73a48f62f352fdb., starting to roll back the global memstore size.
    java.lang.IllegalStateException: Could not instantiate a region instance.
    =====Caused by: java.lang.reflect.InvocationTargetException
    =====Caused by: java.lang.IllegalArgumentException: Need table descriptor

## 解决hbase报错：(同版本的CDH数据转移,只需转移/hbase/data/default, 然后删除并重建本地/hbaes目录) ##

*  一般情况, hbck 都能自动修复

su hbase
    hbase hbck -fix -repair 表名
    #再不行的话就先禁用 hbase该表，然后再次执行此命令

*  实在都不行以后，采用下面的 "暴力还原"方式

su hdfs
    #1, 备份hdfs的表数据
    hdfs dfs -mv /hbase/data/default  /hbase.back
    #2, 删除hdfs中的hbase数据目录的全部数据
    hdfs dfs -rmr /hbase
    #3, 恢复hbase表数据 
    hdfs dfs -mkdir -p /hbase/data/ && hdfs dfs -mv /hbase.back  /hbase/data/default && hdfs dfs -chown -R hbase:hbase /hbase
    
    #4, 删除zk中的hbaes元数据, 防止region分不出去
    echo "rmr /hbase" | zookeeper-client
    
    #5, 重启hbase集群: 开始恢复hbase元数据 
    exit
    sed -i  "s@hbase:/bin/false@hbase:/bin/bash@" /etc/passwd
    cat /etc/passwd |grep hbase
    #hbase:x:484:483:HBase:/var/lib/hbase:/bin/bash
    su hbase #然后重启hbase集群，之后执行hbase hbck -repair #可以指定具体表名: -repair 't1'

*  DoNotRetryIOException  
    解决：排查是因为：/tmp/hbase-hbase/目录的权限非hbase用户所有　—>更改目录或更改此目录权限

Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.DoNotRetryIOException): org.apache.hadoop.hbase.DoNotRetryIOException:
    Mkdirs failed to create /tmp/hbase-hbase/local/jars/tmp (exists=false, cwd=file:/opt/cm-5.12.2/run/cloudera-scm-agent/process/172-hbase-MASTER) 
    Set hbase.table.sanity.checks to false at conf or table descriptor if you want to bypass sanity checks

# 2，hbase表：数据结构 #

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70]

## 1， 如何查看hbase表的region ##

*  1，hmaster 的webui: http://hmaster主机名:60010/master-status
 *  2，hbase(main):044:0> scan ‘hbase:meta’
 *  3，hdfs dfs -ls /hbase/data/default/表名

插入hbase数据，手动flush，把memcache刷写到磁盘形成hfile文件

hbase(main):020:0> put 'test','row98','f:name','bb13'
    0 row(s) in 0.0100 seconds
    
    hbase(main):022:0> flush 'test'
    0 row(s) in 0.3770 seconds

在hdfs：50070界面找到hbase表的目录，命令行查看hfile大小

hdfs dfs -ls -h /hbase/data/default/test/b28cc0d4e59983fbf9d698689fdd4cf7/f  		
    	
    -rw-r--r--	hbase	hbase	1.25 KB	Fri Mar 08 10:28:07 +0800 2019	3	128 MB	5a9c71e3e4854d398976119c135a6043
    -rw-r--r--	hbase	hbase	1021 B	Fri Mar 08 10:22:33 +0800 2019	3	128 MB	895eb2a3bdc14d0cac6dd35cb2ce3a3a
    -rw-r--r--	hbase	hbase	1023 B	Fri Mar 08 03:22:45 +0800 2019	3	128 MB	a29029332d5c416fb2c17f035611a96b

## 2, hfile小文件太多，会影响io读写效率，需要合并 ##

hbase(main):023:0> major_compact 'test'
    0 row(s) in 0.1020 seconds
    
    -rw-r--r--   3 hbase hbase      1.3 K 2019-03-08 10:41 /hbase/data/default/test/b28cc0d4e59983fbf9d698689fdd4cf7/f/8696f1c63ea94f19a387cf3ab4046ad1

## 3, region太大，需要手动切分 ##

hbase(main):031:0> scan 'test'
    ROW                                  COLUMN+CELL                                                                                               
     row1                                column=f:name, timestamp=1551982669892, value=lisi                                                        
     row10                               column=f:name, timestamp=1552012082265, value=bb15                                                        
     row11                               column=f:name, timestamp=1552015957267, value=bb22                                                        
     row12                               column=f:name, timestamp=1552015962717, value=bb23                                                        
     row2                                column=f:name, timestamp=1552011742484, value=aa                                                          
     row3                                column=f:name, timestamp=1552012033034, value=bb                                                          
     row4                                column=f:name, timestamp=1552012038668, value=bb3                                                         
     row5                                column=f:name, timestamp=1552012043951, value=bb5                                                         
     row6                                column=f:name, timestamp=1552012054291, value=bb7                                                         
     row7                                column=f:name, timestamp=1552012061733, value=bb9                                                         
     row8                                column=f:name, timestamp=1552012069224, value=bb11                                                        
     row98                               column=f:name, timestamp=1552012075637, value=bb13                                                        
    12 row(s) in 0.3550 seconds
    
    # 切分大的region,查看hdfs中的region分布
    hbase(main):030:0> split 'test','3'
    0 row(s) in 0.0680 seconds
    ------
    [xinhua@YHEMR-APP03 ~]$ hdfs dfs -ls /hbase/data/default/test/
    drwxr-xr-x   - hbase hbase          0 2019-03-08 11:34 /hbase/data/default/test/a0324435468a3c00fa2f59d9ef2c43b5
    drwxr-xr-x   - hbase hbase          0 2019-03-08 11:34 /hbase/data/default/test/b28cc0d4e59983fbf9d698689fdd4cf7
    drwxr-xr-x   - hbase hbase          0 2019-03-08 11:34 /hbase/data/default/test/f977a4f0c1998a21f142b7ca86ff1189
    
    #major_compact, 删除旧的数据
    hbase(main):032:0> major_compact 'test'
    0 row(s) in 0.0690 seconds
    ------
    [xinhua@YHEMR-APP03 ~]$ hdfs dfs -ls /hbase/data/default/test/
    drwxr-xr-x   - hbase hbase          0 2019-03-08 11:34 /hbase/data/default/test/a0324435468a3c00fa2f59d9ef2c43b5
    drwxr-xr-x   - hbase hbase          0 2019-03-08 11:34 /hbase/data/default/test/f977a4f0c1998a21f142b7ca86ff1189

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70]: /images/20220311/d2e82896db2744a2baabf22912722d23.png