Disabling Kerberos for CDH

给我写信
GitHub

Kerberos

参考：
+ Disabling Kerberos
+ Disabling Kerberos Authentication
+ zookeeper can't delete directory
+

After installing Kerberos for CDH cluster, we may want to disable Kerberos at some point.

HDFS

Change the following configurations on Cloudera Manager web:

hadoop.security.authentication -> Simple
hadoop.security.authorization -> false
dfs.datanode.address -> from 1004 (for Kerberos) to 50010 (default)
dfs.datanode.http.address -> from 1006 (for Kerberos) to 50075 (default)

ZooKeeper

HBase

Change the following configurations:

hbase.security.authentication -> simple

重启，发现无法启动HMaster，其他role都能正常启动。查看HMaster的日志：

master:60000-0x24fff8c28c60006, quorum=hadoop1.com:2181,hadoop4.com:2181,hadoop5.com:2181, baseZNode=/hbase Unable to get data of znode /hbase/splitWAL/WALs%2Fhadoop3.com%2C60020%2C1442886930815-splitting%2Fhadoop3.com%252C60020%252C1442886930815.1442886937853.meta
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /hbase/splitWAL/WALs%2Fhadoop3.com%2C60020%2C1442886930815-splitting%2Fhadoop3.com%252C60020%252C1442886930815.1442886937853.meta
...
master:60000-0x24fff8c28c60006, quorum=hadoop1.com:2181,hadoop4.com:2181,hadoop5.com:2181, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /hbase/splitWAL/WALs%2Fhadoop3.com%2C60020%2C1442886930815-splitting%2Fhadoop3.com%252C60020%252C1442886930815.1442886937853.meta
...
Unhandled exception. Starting shutdown.
java.io.IOException: org.apache.zookeeper.KeeperException\$NoAuthException: KeeperErrorCode = NoAuth for /hbase/splitWAL/WALs%2Fhadoop3.com%2C60020%2C1442886930815-splitting%2Fhadoop3.com%252C60020%252C1442886930815.1442886937853.meta

重点看这句
"baseZNode=/hbase Unable to get data of znode /hbase/splitWAL/WALs%2Fhadoop3.com%2C60020%2C1442886930815-splitting%2Fhadoop3.com%252C60020%252C1442886930815.1442886937853.meta"

关于ZooKeeper中的SASL Authentication，参考 SASL Authentication with ZooKeeper

我们来看看这个znode的ACL状况（关于znode在HBase中的作用，可以参考What are HBase znodes?）

hbase zkcli
$ getAcl /hbase/splitWAL/WALs%2Fhadoop3.com%2C60020%2C1442886930815-splitting%2Fhadoop3.com%252C60020%252C1442886930815.1442886937853.meta
'sasl,'hbase
: cdrwa

为什么会有NoAuthException的错误？
这里的'sasl,'hbase是什么意思？

Generally, you may see such errors when the cluster was not unsecured properly. When the cluster is secured, zookeeper uses SASL authentication and SASL will be the authorization scheme to access the znode data. However, if the authorization scheme is not set to "world:anyone:cdrwa" and the cluster is unsecured, the nodes will still have SASL authorization scheme, but since SASL is not used on secured hbase, "No Auth" or "Authentication is not valid" error will be seen.

那么禁用Kerberos后，还需要对znode做什么处理呢？看Cloudera Community中的一个回帖：

backing out kerberos is not an automatic process currently as there can be many services using Zookeeper and it retains those ACLs which were set while kerberos was enabled. We have developed a little java program for our customers that backs out the ACLs from ZK, but all it really does is iterate over all the znodes in /hbase and set their acls to world:anyone.

So, you can just manually do this as well. This is an example:

setAcl /hbase world:anyone:cdrwa

You would need to do that on every znode under /hbase and the master will start.

关于znode的ACL设置，参考Apache ZooKeeper – Setting ACL in ZooKeeper Client

我想把这个znode的ACL改为world:anyone:cdrwa

[root@hadoop1 ~]# sudo -u hbase hbase zkcli

$ ls /hbase/splitWAL
[WALs%2Fhadoop3.com%2C60020%2C1442886930815-splitting%2Fhadoop3.com%252C60020%252C1442886930815.1442886937853.meta]

$ getAcl /hbase/splitWAL/WALs%2Fhadoop3.com%2C60020%2C1442886930815-splitting%2Fhadoop3.com%252C60020%252C1442886930815.1442886937853.meta
'sasl,'hbase
: cdrwa

$ setAcl /hbase/splitWAL/WALs%2Fhadoop3.com%2C60020%2C1442886930815-splitting%2Fhadoop3.com%252C60020%252C1442886930815.1442886937853.meta world:anyone:cdrwa
Authentication is not valid

通过setAcl命令来修改该znode的ACL权限失败。我们可以通过super user这个身份，在绕过ACL限制的情况下，将该znode的ACL设置为world:anyone:cdrwa。

1. 假设super user的用户名为super，其密码明文为super-1234，用命令计算其用户密码的base64密文：

   [root@hadoop5 ~]# java -cp   
   /opt/cloudera/parcels/CDH-5.3.2-1.cdh5.3.2.p0.10/jars/zookeeper-3.4.5-cdh5.3.2.jar:/usr/share/cmf/lib/slf4j-api-1.6.1.jar org.apache.zookeeper.server.auth.DigestAuthenticationProvider super:super-1234
   SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
   SLF4J: Defaulting to no-operation (NOP) logger implementation
   SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
   super:super-1234->super:noK4qI8mmROFXyywTCCvZ62P7xs=

   可见，产生的密码的密文是noK4qI8mmROFXyywTCCvZ62P7xs=

2. 在每一个ZooKeeper节点上，在ZooKeeper的启动脚本（/usr/lib64/cmf/service/zookeeper/zkserver.sh）中添加一个JVM参数export ZOOKEEPER_SERVER_OPTS="-Dzookeeper.DigestAuthenticationProvider.superDigest=super:noK4qI8mmROFXyywTCCvZ62P7xs="
3. 重启ZooKeeper集群

4. 用命令/opt/cloudera/parcels/CDH-5.3.2-1.cdh5.3.2.p0.10/lib/zookeeper/bin/zkCli.sh进入ZooKeeper的客户端，然后

   addauth digest super:super-1234
   上面一句话的目的是切换到super user的身份
   setAcl /hbase/splitWAL/WALs%2Fhadoop3.com%2C60020%2C1442886930815-s plitting%2Fhadoop3.com%252C60020%252C1442886930815.1442886937853.meta world:anyone:cdrwa

   现在就不会报错了，核实一下

   getAcl /hbase/splitWAL/WALs%2Fhadoop3.com%2C60020%2C1442886930815-splitting%2Fhadoop3.com%252C60020%252C1442886930815.1442886937853.meta     
   'world,'anyone
   : cdrwa

后遗症：这样设置了super user后，Cloudera Manager中总是会提示ZooKeeper不正常（Canery Test失败），猜测愿意是Canery Test需要以某个super user的身份来进行该项测试，而我们却把super user的密码设置为了一个其他值。