HBase运维问题

给我写信
GitHub

HBase

参考

• Checking and Repairing HBase Tables
• HBase - Region in Transition

找不到所有的Meta Data

发生场景：

为CDH集群enable Kerberos，后来发生问题又disable Kerberos了。期间HBase/ZK发生了一个小问题，通过这个方法修复了。（见这里的HBase这一节的描述）。之后，HBase可以正常启动，用list也可以展示所有的table，但是无法scan任何一个table，也不能创建table。用sudo -u hbase hbase hbck检查时输出如下内容：

Version: 0.98.6-cdh5.3.2
Number of live region servers: 5
Number of dead region servers: 0
Master: hadoop2.com,60000,1444621090883
Number of backup masters: 0
Average load: 0.0
Number of requests: 2
Number of regions: 0
Number of regions in transition: 1
ERROR: META region or some of its attributes are null.
ERROR: hbase:meta is not found on any region.
ERROR: hbase:meta table is not consistent. Run HBCK with proper fix options to fix hbase:meta inconsistency. Exiting...
15/10/12 14:04:50 INFO Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
15/10/12 14:04:56 INFO zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=hadoop1.com:2181,hadoop4.com:2181,hadoop5.com:2181
15/10/12 14:04:56 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop1.com:2181,hadoop4.com:2181,hadoop5.com:2181 sessionTimeout=60000 watcher=hbase Fsck, quorum=hadoop1.com:2181,hadoop4.com:2181,hadoop5.com:2181, baseZNode=/hbase
15/10/12 14:04:56 INFO zookeeper.ClientCnxn: Opening socket connection to server hadoop5.com/59.215.222.76:2181. Will not attempt to authenticate using SASL (unknown error)
15/10/12 14:04:56 INFO zookeeper.ClientCnxn: Socket connection established to hadoop5.com/59.215.222.76:2181, initiating session
15/10/12 14:04:56 INFO zookeeper.ClientCnxn: Session establishment complete on server hadoop5.com/59.215.222.76:2181, sessionid = 0x35059e649fa0160, negotiated timeout = 60000
Summary:
3 inconsistencies detected.
Status: INCONSISTENT

分析：
看来问题在于元数据，关于HBase的metadata，参见 Catalog Tables 以及 HBase .META. Layout。

既然MetaData出现了问题，那为什么hbase shell中的list可以列出所有的tables呢？个人猜想这些表名数据是从ZK中取得的。通过 hbase zkcli 中的命令ls /hbase/table可以验证。

解决方法：
根据 HBase - Region in Transition:
1. Stop HBase
2. Use a zookeeper cli such as "hbase zkcli" and run "rmr /hbase" to delete the HBase znodes
3. Restart HBase. It will recreate the znodes

Regions Stuck in RIT / Region Not Deployed on any RS

发生场景：

通过hbck命令检查HBase的状态，发现异常。

$ sudo -u hbase hbase hbck
··· ···
ERROR: Region { meta => clone-QSH_S_METER_DATA,\xF8,1422783824955.1dfc13d0ea5596e76a3d1c3baadf6334., hdfs => hdfs://nameservice1/hbase/data/default/clone-QSH_S_METER_DATA/1dfc13d0ea5596e76a3d1c3baadf6334, deployed => } not deployed on any region server.
ERROR: Region { meta => clone-QSH_S_METER_DATA,\xD8,1422783824955.3fe03bf15c1f3ac75647c8fad947d489., hdfs => hdfs://nameservice1/hbase/data/default/clone-QSH_S_METER_DATA/3fe03bf15c1f3ac75647c8fad947d489, deployed => } not deployed on any region server.
··· ···
··· ···
ERROR: Region { meta => clone-QSH_S_METER_DATA,\xB8,1422783824955.f3d97f63a783410160c687b13174583e., hdfs => hdfs://nameservice1/hbase/data/default/clone-QSH_S_METER_DATA/f3d97f63a783410160c687b13174583e, deployed => } not deployed on any region server.
ERROR: There is a hole in the region chain between \x90 and \xA8. You need to create a new .regioninfo and region dir in hdfs to plug the hole.
ERROR: There is a hole in the region chain between \xB0 and \xC8. You need to create a new .regioninfo and region dir in hdfs to plug the hole.
ERROR: There is a hole in the region chain between \xD0 and \xE8. You need to create a new .regioninfo and region dir in hdfs to plug the hole.
ERROR: Last region should end with an empty key. You need to create a new region and regioninfo in HDFS to plug the hole.
15/10/28 09:08:44 INFO util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
ERROR: Found inconsistency in table clone-QSH_S_METER_DATA
Summary:
Table clone-QSH_S_METER_DATA is inconsistent.
Number of regions: 21
Deployed on: ecs1.njzd.com,60020,1444035878618 ecs3.njzd.com,60020,1444035879123 ecs4.njzd.com,60020,1444035879838 ecs5.njzd.com,60020,1444035881022
··· ···
15 inconsistencies detected.
Status: INCONSISTENT

通过HBase的WEB，可以看到有些region一直处于RIT的状态，如下：

试图通过命令hbase hbck -repair或者hbase hbck -fixMeta -fixAssignments来修复，均失败了。

[root@ecs3 ~]# sudo -u hbase hbase hbck -repair
··· ···
15/10/28 09:05:22 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 1dfc13d0ea5596e76a3d1c3baadf6334, NAME => 'clone-QSH_S_METER_DATA,\xF8,1422783824955.1dfc13d0ea5596e76a3d1c3baadf6334.', STARTKEY => '\xF8', ENDKEY => ''}
15/10/28 09:05:23 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 1dfc13d0ea5596e76a3d1c3baadf6334, NAME => 'clone-QSH_S_METER_DATA,\xF8,1422783824955.1dfc13d0ea5596e76a3d1c3baadf6334.', STARTKEY => '\xF8', ENDKEY => ''}
··· ···
Exception in thread "main" java.io.IOException: Region {ENCODED => 1dfc13d0ea5596e76a3d1c3baadf6334, NAME => 'clone-QSH_S_METER_DATA,\xF8,1422783824955.1dfc13d0ea5596e76a3d1c3baadf6334.', STARTKEY => '\xF8', ENDKEY => ''} failed to move out of transition within timeout 120000ms
    at org.apache.hadoop.hbase.util.HBaseFsckRepair.waitUntilAssigned(HBaseFsckRepair.java:139)
    at org.apache.hadoop.hbase.util.HBaseFsck.tryAssignmentRepair(HBaseFsck.java:1732)
    at org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:1873)
    at org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixConsistency(HBaseFsck.java:1559)
    at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:465)
    at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:484)
    at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:4032)
    at org.apache.hadoop.hbase.util.HBaseFsck$HBaseFsckTool.run(HBaseFsck.java:3841)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
    at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3829)

分析

# 1. 什么是RIT状态？

As regions are managed by the master and region servers to, for example, balance the load across servers, they go through short phases of transition. This applies to opening, closing, and splitting a region. Before the operation is performed, the region is added to the "Regions in Transition" list on the WEB UI, and once the operation is complete, it is removed.

只要有region处于RIT状态，balacer就无法运行。

# 2.一个Region可能处于哪几种状态？
Region的状态由master跟踪，包括以下状态：

Region在这些状态之间的迁移（transition）可以由master引发，也可以由region server引发。

# 3. 为什么有些Region会一直处于RIT状态？
可能的原因：
a) 该region被原来的RS unassigned了，但是还没有被assigned到一个新的RS上，处于无主状态。

综合分析
从命令hbase hbck -repair的输出来看，一些region并没有于assigned的状态（例如clone-QSH_S_METER_DATA,\xF8,1422783824955.1dfc13d0ea5596e76a3d1c3baadf6334.），实际上，从HBase web可以看出，它们处于FAILED_OPEN的状态。

这个问题参考 HBASE-12480

我尝试过HBase shell里的几个命令：move, assign, unassign, close_region，但是都不行。甚至重启了HBased集群，问题依然存在。

通过查看clone-QSH_S_METER_DATA,\xF8,1422783824955.1dfc13d0ea5596e76a3d1c3baadf6334.所在的RS的日志，找出为什么这个region无法打开：

Failed open of region=clone-QSH_S_METER_DATA,\xF8,1422783824955.1dfc13d0ea5596e76a3d1c3baadf6334., starting to roll back the global memstore size.
java.io.IOException: java.io.IOException: java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://nameservice1/hbase/data/default/QSH_S_METER_DATA/b779f6b18dd95bf51662c07a00d781e5/F/827fb1148dd54bef8dfe61a093022380, hdfs://nameservice1/hbase/.tmp/data/default/QSH_S_METER_DATA/b779f6b18dd95bf51662c07a00d781e5/F/827fb1148dd54bef8dfe61a093022380, hdfs://nameservice1/hbase/archive/data/default/QSH_S_METER_DATA/b779f6b18dd95bf51662c07a00d781e5/F/827fb1148dd54bef8dfe61a093022380]
at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionStores(HRegion.java:805)
at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:716)
...
...

Caused by: java.io.IOException: java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://nameservice1/hbase/data/default/QSH_S_METER_DATA/b779f6b18dd95bf51662c07a00d781e5/F/827fb1148dd54bef8dfe61a093022380, hdfs://nameservice1/hbase/.tmp/data/default/QSH_S_METER_DATA/b779f6b18dd95bf51662c07a00d781e5/F/827fb1148dd54bef8dfe61a093022380, hdfs://nameservice1/hbase/archive/data/default/QSH_S_METER_DATA/b779f6b18dd95bf51662c07a00d781e5/F/827fb1148dd54bef8dfe61a093022380]
at org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:557)
...
...

Caused by: java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://nameservice1/hbase/data/default/QSH_S_METER_DATA/b779f6b18dd95bf51662c07a00d781e5/F/827fb1148dd54bef8dfe61a093022380, hdfs://nameservice1/hbase/.tmp/data/default/QSH_S_METER_DATA/b779f6b18dd95bf51662c07a00d781e5/F/827fb1148dd54bef8dfe61a093022380, hdfs://nameservice1/hbase/archive/data/default/QSH_S_METER_DATA/b779f6b18dd95bf51662c07a00d781e5/F/827fb1148dd54bef8dfe61a093022380]
at org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.tryOpen(FileLink.java:305)
...
...

原因很明显：这个region在HDFS上的对应的HFile文件不存在了。为什么会不存在呢？对于本案例而言，clone-QSH_S_METER_DATA是由QSH_S_METER_DATA克隆而来的，所以使用的是QSH_S_METER_DATA的HFile文件的link。然而，QSH_S_METER_DATA的HFiles发生了变化（split/merge等），而clone-QSH_S_METER_DATA对此却毫不知情，因此找不到对应的HDFS文件。

这是猜测，但是从HBase web的Regions in Transition可以看到，所有处于RIT的regions都属于clone-QSH_S_METER_DATA。