[关闭]
@aitanjupt 2015-10-18T19:16:12.000000Z 字数 8883 阅读 2910

A record--Deployment of Big Data Platform CDH Cluste

Cloudera-Manager CDH Hadoop Deploy Cluster

Abstract: Deployment and Management of Hadoop clusters need tools, such as Cloudera Manager. In this article, I compare the tools briefly, and then record the step of deploying CDH cluster offline in detail. Finally, I expound the theory of 'handle delicately'.

Preface

The emergence of Big Data technology led by Apache Hadoop, makes small and medium-sized enterprises also have the ability to handle the storage and processing of big data. At present, there'r lots of Hadoop distributions, such as HUAWEI Distribution, Intel Distribution, Cloudera’s Distribution Including Apache Hadoop (CDH free), and Hortonworks Data Platform (HDP free), etc. All of these are based on the Hadoop Apache Community Edition.

The deployment and management of a Hadoop cluster which has tens or more nodes needs advanced tools. Apache Ambari from Hortonworks is this kind of tools, it provided an easy-to-use RESTfull web site to manage Hadoop. Cloudera also provided a similar tool, Cloudera Manager(CM) to configure, monitor and manage CDH clusters.

The main content of this paper is a record of building a CDH cluster. Special attention is required to choose Cloudera Manager version, which depends on Operating system, el7 isn't supported by Cloudera Manager at this moment. You should follow the official document, otherwise the installation will run into a stone wall.

This paper is based on CentOS 6.5, 64-bit；Cloudera Manager 5.3.6；JDK 1.7.

Deploy CDH

Configure network (All nodes)

  1. [root@cdh-server ~]# vi /etc/sysconfig/network #修改hostname:
  2. NETWORKING=yes
  3. HOSTNAME=cdh-server
  5. [root@cdh-server ~]# vi /etc/hosts #修改ip与主机名的对应关系:
  6. 192.168.180.173 cdh-server
  7. 192.168.180.175 node175
  8. [root@cdh-server ~]# service network restart #重启网络服务生效

Install JDK (All nodes)

  1. #卸载OpenJDK
  2. [root@cdh-server user1]# rpm -qa | grep java
  3. [root@cdh-server user1]# rpm -e --nodeps java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64
  4. [root@cdh-server user1]# rpm -e --nodeps java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_64
  5. [root@cdh-server user1]# rpm -e --nodeps java-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64
  6. #安装JDK
  7. [root@cdh-server user1]# chmod a+x jdk-7u79-linux-x64.rpm
  8. [root@cdh-server user1]# rpm -ivh jdk-7u79-linux-x64.rpm
  9. [root@cdh-server user1]# echo "JAVA_HOME=/usr/java/jdk1.7.0_79/" >>

Install MySQL (Master)

  1. [user1@cdh-server]$ cd /home/user1
  2. [user1@cdh-server]$ tar -zxvf mysql-5.6.26-linux-glibc2.5-x86_64.tar.gz
  3. [user1@cdh-server]$ mv mysql-5.6.26-linux-glibc2.5-x86_64 mysql-5.6.26
  4. [user1@cdh-server]$ cd mysql-5.6.26/
  5. [user1@cdh-server]$ vi support-files/my.cnf    #新建文件
  1. +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  2. [mysqld]
  3. character-set-server=utf8
  4. default-storage-engine=INNODB
  6. # Uncomment the following if you are using InnoDB tables
  7. innodb_data_home_dir = /home/user1/mysql-5.6.26/data
  8. innodb_data_file_path = ibdata1:10M:autoextend
  9. innodb_log_group_home_dir = /home/user1/mysql-5.6.26/data
  10. # You can set .._buffer_pool_size up to 50 - 80 %
  11. # of RAM but beware of setting memory usage too high
  12. innodb_buffer_pool_size = 16M
  13. innodb_additional_mem_pool_size = 2M
  14. # Set .._log_file_size to 25 % of buffer pool size
  15. innodb_log_file_size = 5M
  16. innodb_log_buffer_size = 8M
  17. innodb_flush_log_at_trx_commit = 1
  18. innodb_lock_wait_timeout = 50
  19. +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Initialize MySQL (Master)

  1. [user1@cdh-server]$ ./scripts/mysql_install_db --defaults-file=/home/user1/mysql-5.6.26/support-files/my.cnf --basedir=/home/user1/mysql-5.6.26 --datadir=/home/user1/mysql-5.6.26/data --user=user1
  2. [user1@cdh-server]$ ./bin/mysqld --defaults-file=/home/user1/mysql-5.6.26/support-files/my.cnf --basedir=/home/user1/mysql-5.6.26 --datadir=/home/user1/mysql-5.6.26/data > mysql.log 2>&1 &
  3. [user1@cdh-server]$ ./bin/mysqladmin -u root password '123456' 
  1. [user1@cdh-server mysql-5.6.26]$ ./bin/mysql -uroot -p'123456'
  2. #hive
  3. mysql> create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
  4. Query OK, 1 row affected (0.00 sec)
  6. #Activity Monitor使用
  7. mysql> create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
  8. Query OK, 1 row affected (0.01 sec)
  10. #Navigator Audit Server使用
  11. mysql> create database audit DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
  12. Query OK, 1 row affected (0.01 sec)
  14. #Navigator Metadata Server
  15. mysql> create database metadata DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
  16. Query OK, 1 row affected (0.01 sec)
  18. mysql> grant all privileges on *.* to 'root'@'localhost' identified by '123456' with grant option;
  19. Query OK, 0 rows affected (0.00 sec)
  21. mysql> grant all privileges on *.* to 'root'@'cdh-server' identified by '123456' with grant option;
  22. Query OK, 0 rows affected (0.00 sec)
  24. #this user scm is for  cloudera manager
  25. mysql> grant all privileges on *.* to 'scm'@'localhost' identified by 'scm' with grant option;
  26. Query OK, 0 rows affected (0.00 sec)
  28. mysql> grant all privileges on *.* to 'scm'@'cdh-server' identified by 'scm' with grant option;
  29. Query OK, 0 rows affected (0.00 sec)
  31. mysql> flush privileges;
  32. Query OK, 0 rows affected (0.00 sec)

Deploy & Start CM-Server (Master)

  1. [user1@cdh-server ~]$ tar -zxvf cloudera-manager-el6-cm5.3.6_x86_64.tar.gz
  2. [user1@cdh-server ~]$ cp mysql-connector-java-5.1.33-bin.jar ./cm-5.3.6/share/cmf/lib/
  4. [user1@cdh-server ~]$ su - root
  5. [root@cdh-server ~]# cd /home/user1/
  7. [root@cdh-server user1]# cp -rf cloudera /opt
  8. [root@cdh-server user1]# mv CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel /opt/cloudera/parcel-repo/CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel
  9. [root@cdh-server user1]# mv CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha /opt/cloudera/parcel-repo/CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha
  10. [root@cdh-server user1]# mv manifest.json /opt/cloudera/parcel-repo/manifest.json
  11. [root@cdh-server user1]# ./cm-5.3.6/share/cmf/schema/scm_prepare_database.sh mysql cm -hlocalhost:3306 -uroot -p123456 --scm-host localhost scm scm scm
  12. [root@cdh-server user1]# ./cm-5.3.6/etc/init.d/cloudera-scm-server start
  13. Starting cloudera-scm-server:                              [  OK  ]
  14. [root@cdh-server user1]# tail -f ./cm-5.3.6/log/cloudera-scm-server/cloudera-scm-server.log

Stop iptables (All nodes)

  1. #停止iptables
  2. [root@cdh-server user1]# service iptables stop 
  3. #通过浏览器访问验证
  4. http://192.168.180.173:7180/

Deploy & Start CM-Agent (Slaves)

  1. [root@cdh-server user1]# tar -zxvf cloudera-manager-el6-cm5.3.6_x86_64.tar.gz
  2. [root@cdh-server user1]# vi cm-5.3.6/etc/cloudera-scm-agent/config.ini
  1. +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  2. # Hostname of the CM server.
  3. #server_host=localhost
  4. server_host=cdh-server
  5. +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  1. [root@cdh-server user1]# useradd -G sys --home=/home/user1/cm-5.3.6/run/cloudera-scm-server --no-create-home  --comment "Cloudera SCM User" cloudera-scm
  2. [root@cdh-server user1]# useradd --comment "Cloudera SCM User" cloudera-scm #若上一步执行正确，则此步省略
  4. [root@cdh-server user1]# echo 0 > /proc/sys/vm/swappiness
  6. [root@cdh-server user1]# ./cm-5.3.6/etc/init.d/cloudera-scm-agent start
  7. Starting cloudera-scm-agent:                               [  OK  ]
  8. [root@cdh-server user1]# tail -f ./cm-5.3.6/log/cloudera-scm-agent/cloudera-scm-agent.log

Configure CDH

Load Cloudera Manager http://192.168.180.173:7180/, then create a new Cluster names Cluster_user1，Start and configure Services.

  1. #安装配置hive出错时，在hiveServer上:
  2. [root@hive-server user1]# cp mysql-connector-java-5.1.33-bin.jar /opt/cloudera/parcels/CDH-5.3.6-1.cdh5.3.6.p0.11/lib/hive/lib/
  4. #同理：use this jar for Navigator Audit Server and Navigator Metadata Server or Activity Server
  5. [root@cdh-server user1]# cp mysql-connector-java-5.1.33-bin.jar /usr/share/java/mysql-connector-java.jar

Others

Stop CDH

  1. [root@cdh-server user1]# ./cm-5.3.6/etc/init.d/cloudera-scm-agent stop
  1. [root@cdh-server user1]# ./cm-5.3.6/etc/init.d/cloudera-scm-server stop

Start CDH

  1. [user1@cdh-server]$ ./bin/mysqld --defaults-file=/home/user1/mysql-5.6.26/support-files/my.cnf --basedir=/home/user1/mysql-5.6.26 --datadir=/home/user1/mysql-5.6.26/data > mysql.log 2>&1 &
  1. [root@cdh-server user1]# ./cm-5.3.6/etc/init.d/cloudera-scm-agent start
  2. Starting cloudera-scm-agent:                               [  OK  ]
  3. [root@cdh-server user1]# tail -f ./cm-5.3.6/log/cloudera-scm-agent/cloudera-scm-agent.log
  1. [root@cdh-server user1]# ./cm-5.3.6/etc/init.d/cloudera-scm-server start
  2. Starting cloudera-scm-server:                              [  OK  ]
  3. [root@cdh-server user1]# tail -f ./cm-5.3.6/log/cloudera-scm-server/cloudera-scm-server.log

The theory of handle delicately

Handle delicately is a kind of feeling, but also a skill. When in seller's market, a company can make a profit while may needn't to handle delicately. However, if it wishes to pursue more, it will handle more delicately, such as pay more attention to detail, User Experience or others. When in buyer's market, the relation between supply and demand makes companies handle delicately to survive.

Handle delicately is not only the driving force pushing social to continually advance, but also the result of this advance. Today, lots of internet firms are in buyer's market, who has more users who will win the fight, delicately handling makes them at the forefront of social evolution and technological innovation.

Handle delicately doesn't only appear in companies, but also individuals, regions and countries, etc. The contry who handles more delicately, who is more developed. The company who pays more attention to details, who is more competitive. But, the one who handles more delicately doesn't mean it will make more profits, for various reasons.

Writer: @Angel Wang
我的头像
aitanjupt@hotmail.com
2015 - 10 - 18

添加新批注
在作者公开此批注前，只有你和作者可见。
回复批注