[关闭]
@nalan90 2017-09-13T14:55:25.000000Z 字数 11236 阅读 635

Hadoop伪分布式配置

大数据

摘自：http://www.powerxing.com/install-hadoop/

核心组件
安装包准备
添加用户并设置权限
  1. ## 添加hadoop用户
  2. [root@dev-162 zhangshuang]# useradd -m hadoop -s /bin/bash
  4. ## 设置密码
  5. [root@dev-162 zhangshuang]# passwd hadoop
  6. Changing password for user hadoop.
  7. New password:
  8. BAD PASSWORD: The password is shorter than 7 characters
  9. Retype new password:
  10. passwd: all authentication tokens updated successfully.
  12. ## 添加sudo权限(追加如下行)
  13. [root@dev-162 zhangshuang]# visudo
  15. hadoop ALL=(ALL) NOPASSWD:ALL
  17. ## 切换至hadoop用户
  18. [root@dev-162 zhangshuang]# su - hadoop
  20. ## 生成ssh证书(全部回车即可)
  21. [hadoop@dev-162 ~]$ ssh-keygen -t rsa
  22. Generating public/private rsa key pair.
  23. Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
  24. Created directory '/home/hadoop/.ssh'.
  25. Enter passphrase (empty for no passphrase):
  26. Enter same passphrase again:
  27. Your identification has been saved in /home/hadoop/.ssh/id_rsa.
  28. Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
  29. The key fingerprint is:
  30. 7a:13:7f:9d:80:37:38:56:e9:b5:de:53:c1:c0:5a:33 hadoop@dev-162
  31. The key randomart image is:
  32. +--[ RSA 2048]----+
  33. |            ..   |
  34. |             Eo  |
  35. |            = +o |
  36. |           * . ..|
  37. |        S = = . .|
  38. |       . + o = o.|
  39. |      . o . . +..|
  40. |       . . .    .|
  41. |                 |
  42. +-----------------+
  44. ## 将公钥写入authorized_keys文件
  45. [hadoop@dev-162 ~]$ cat .ssh/id_rsa.pub >> .ssh/authorized_keys
  47. ## 修改权限
  48. [hadoop@dev-162 ~]$ chmod 600 .ssh/authorized_keys
  50. ## 无密码登录本机
  51. [hadoop@dev-162 ~]$ ssh localhost
  52. The authenticity of host 'localhost (::1)' can not be established.
  53. ECDSA key fingerprint is fe:0f:40:d7:56:c8:c1:b4:29:c3:ce:d8:d6:12:66:2e.
  54. Are you sure you want to continue connecting (yes/no)? yes
  55. Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
  56. Last login: Wed Aug 23 13:56:50 2017
安装java环境
  1. ## 安装openjdk
  3. [hadoop@dev-162 ~]$ sudo yum -y install java-1.8.0-openjdk
  4. [hadoop@dev-162 ~]$ sudo yum -y install java-1.8.0-openjdk-devel
  6. ## 设置java环境变量 (注意是.bashrc不是.bash_profile)
  7. [hadoop@dev-161 ~]$ vim .bashrc
  8. ## 追加如下行
  9. export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
  10. export PATH=$JAVA_HOME/bin:$PATH:/usr/local/hadoop/sbin:/usr/local/hadoop/bin
  11. export HADOOP_HOME=/usr/local/hadoop
  12. export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
  14. [hadoop@dev-161 ~]$ source .bashrc
  15. [hadoop@dev-161 ~]$ echo $JAVA_HOME
  16. /usr/lib/jvm/java-1.8.0-openjdk
  17. [hadoop@dev-161 ~]$ java -version
  18. openjdk version "1.8.0_141"
  19. OpenJDK Runtime Environment (build 1.8.0_141-b16)
  20. OpenJDK 64-Bit Server VM (build 25.141-b16, mixed mode)
  22. [hadoop@dev-161 ~]$ $JAVA_HOME/bin/java -version
  23. openjdk version "1.8.0_141"
  24. OpenJDK Runtime Environment (build 1.8.0_141-b16)
  25. OpenJDK 64-Bit Server VM (build 25.141-b16, mixed mode)
下载hadoop
  1. ## download hadoop-2.7.4.tar.gz
  2. [hadoop@dev-162 ~]$ wget https://mirrors.cnnic.cn/apache/hadoop/common/hadoop-2.7.4/hadoop-2.7.4.tar.gz --no-check-certificate
  4. ## tar hadoop-2.7.4.tar.gz to /usr/local
  5. [hadoop@dev-161 tmp]$ sudo tar -zxvf hadoop-2.7.4.tar.gz -C /usr/local/
  6. [hadoop@dev-161 tmp]$ cd /usr/local/
  8. ## rename /usr/local/hadoop-2.7.4 dir to /usr/local/hadoop
  9. [hadoop@dev-161 local]$ sudo mv hadoop-2.7.4/ hadoop
  11. ## change /usr/local/hadoop dir owner
  12. [hadoop@dev-161 local]$ sudo chown -R hadoop hadoop/
  13. [hadoop@dev-161 local]$ cd /usr/local/hadoop/
  15. ## show hadoop version
  16. [hadoop@dev-161 hadoop]$ ./bin/hadoop version
  17. Hadoop 2.7.4
  18. Subversion https://shv@git-wip-us.apache.org/repos/asf/hadoop.git -r cd915e1e8d9d0131462a0b7301586c175728a282
  19. Compiled by kshvachk on 2017-08-01T00:29Z
  20. Compiled with protoc 2.5.0
  21. From source with checksum 50b0468318b4ce9bd24dc467b7ce1148
  22. This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-2.7.4.jar
配置core-site.xml
  1. ## 配置core-site.xml
  2. [hadoop@dev-161 hadoop]$ pwd
  3. /usr/local/hadoop
  4. [hadoop@dev-161 hadoop]$ vim etc/hadoop/core-site.xml
  6. <configuration>
  7.         <property>
  8.              <name>hadoop.tmp.dir</name>
  9.              <value>file:/usr/local/hadoop/tmp</value>
  10.              <description>Abase for other temporary directories.</description>
  11.         </property>
  12.         <property>
  13.              <name>fs.defaultFS</name>
  14.              <value>hdfs://localhost:9000</value>
  15.         </property>
  16. </configuration>
配置hdfs-site.xml
  1. ## 配置 hdfs-site.xml
  2. [hadoop@dev-161 hadoop]$ vim etc/hadoop/hdfs-site.xml
  3. <configuration>
  4.         <property>
  5.              <name>dfs.replication</name>
  6.              <value>1</value>
  7.         </property>
  8.         <property>
  9.              <name>dfs.namenode.name.dir</name>
  10.              <value>file:/usr/local/hadoop/tmp/dfs/name</value>
  11.         </property>
  12.         <property>
  13.              <name>dfs.datanode.data.dir</name>
  14.              <value>file:/usr/local/hadoop/tmp/dfs/data</value>
  15.         </property>
  16. </configuration>
格式化并启动hdfs
  1. [hadoop@dev-161 hadoop]$ ./bin/hdfs namenode -format
  3. ## 出现以下信息表示成功
  4. 17/08/23 14:21:21 INFO common.Storage: Storage directory /usr/local/hadoop/tmp/dfs/name has been successfully formatted.
  5. 17/08/23 14:21:21 INFO util.ExitUtil: Exiting with status 0
  7. [hadoop@dev-161 ~]$ cd /usr/local/hadoop/
  9. ## 启动hdfs
  10. [hadoop@dev-161 hadoop]$ ./sbin/start-dfs.sh
  11. Starting namenodes on [localhost]
  12. localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-namenode-dev-161.out
  13. localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hadoop-datanode-dev-161.out
  14. Starting secondary namenodes [0.0.0.0]
  15. The authenticity of host '0.0.0.0 (0.0.0.0)' can not be established.
  16. ECDSA key fingerprint is fe:0f:40:d7:56:c8:c1:b4:29:c3:ce:d8:d6:12:66:2e.
  17. Are you sure you want to continue connecting (yes/no)? yes
  18. 0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.
  19. 0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-secondarynamenode-dev-161.out
  21. ## 启动成功
  22. [hadoop@dev-161 hadoop]$ jps
  23. 5714 DataNode
  24. 5589 NameNode
  25. 6022 Jps
  26. 5881 SecondaryNameNode
  28. # webui
  29. http://172.16.1.161:50070/dfshealth.html#tab-overview

image_1bo6st27211a0cgu1occ11iu1t6u9.png-184.5kB

hdfs常用命令
  1. [hadoop@dev-161 hadoop]$ ./bin/hdfs
  2. Usage: hdfs [--config confdir] [--loglevel loglevel] COMMAND
  3.        where COMMAND is one of:
  4.   dfs                  run a filesystem command on the file systems supported in Hadoop.
  5.   classpath            prints the classpath
  6.   namenode -format     format the DFS filesystem
  7.   secondarynamenode    run the DFS secondary namenode
  8.   namenode             run the DFS namenode
  9.   journalnode          run the DFS journalnode
  10.   zkfc                 run the ZK Failover Controller daemon
  11.   datanode             run a DFS datanode
  12.   dfsadmin             run a DFS admin client
  13.   haadmin              run a DFS HA admin client
  14.   fsck                 run a DFS filesystem checking utility
  15.   balancer             run a cluster balancing utility
  16.   jmxget               get JMX exported values from NameNode or DataNode.
  17.   mover                run a utility to move block replicas across
  18.                        storage types
  19.   oiv                  apply the offline fsimage viewer to an fsimage
  20.   oiv_legacy           apply the offline fsimage viewer to an legacy fsimage
  21.   oev                  apply the offline edits viewer to an edits file
  22.   fetchdt              fetch a delegation token from the NameNode
  23.   getconf              get config values from configuration
  24.   groups               get the groups which users belong to
  25.   snapshotDiff         diff two snapshots of a directory or diff the
  26.                        current directory contents with a snapshot
  27.   lsSnapshottableDir   list all snapshottable dirs owned by the current user
  28.                         Use -help to see options
  29.   portmap              run a portmap service
  30.   nfs3                 run an NFS version 3 gateway
  31.   cacheadmin           configure the HDFS cache
  32.   crypto               configure HDFS encryption zones
  33.   storagepolicies      list/get/set block storage policies
  34.   version              print the version
  36. Most commands print help when invoked w/o parameters.
  38. ----------
  40. [hadoop@dev-161 hadoop]$ ./bin/hdfs dfs
  41. Usage: hadoop fs [generic options]
  42.     [-appendToFile <localsrc> ... <dst>]
  43.     [-cat [-ignoreCrc] <src> ...]
  44.     [-checksum <src> ...]
  45.     [-chgrp [-R] GROUP PATH...]
  46.     [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
  47.     [-chown [-R] [OWNER][:[GROUP]] PATH...]
  48.     [-copyFromLocal [-f] [-p] [-l] <localsrc> ... <dst>]
  49.     [-copyToLocal [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
  50.     [-count [-q] [-h] <path> ...]
  51.     [-cp [-f] [-p | -p[topax]] <src> ... <dst>]
  52.     [-createSnapshot <snapshotDir> [<snapshotName>]]
  53.     [-deleteSnapshot <snapshotDir> <snapshotName>]
  54.     [-df [-h] [<path> ...]]
  55.     [-du [-s] [-h] <path> ...]
  56.     [-expunge]
  57.     [-find <path> ... <expression> ...]
  58.     [-get [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
  59.     [-getfacl [-R] <path>]
  60.     [-getfattr [-R] {-n name | -d} [-e en] <path>]
  61.     [-getmerge [-nl] <src> <localdst>]
  62.     [-help [cmd ...]]
  63.     [-ls [-d] [-h] [-R] [<path> ...]]
  64.     [-mkdir [-p] <path> ...]
  65.     [-moveFromLocal <localsrc> ... <dst>]
  66.     [-moveToLocal <src> <localdst>]
  67.     [-mv <src> ... <dst>]
  68.     [-put [-f] [-p] [-l] <localsrc> ... <dst>]
  69.     [-renameSnapshot <snapshotDir> <oldName> <newName>]
  70.     [-rm [-f] [-r|-R] [-skipTrash] <src> ...]
  71.     [-rmdir [--ignore-fail-on-non-empty] <dir> ...]
  72.     [-setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]]
  73.     [-setfattr {-n name [-v value] | -x name} <path>]
  74.     [-setrep [-R] [-w] <rep> <path> ...]
  75.     [-stat [format] <path> ...]
  76.     [-tail [-f] <file>]
  77.     [-test -[defsz] <path>]
  78.     [-text [-ignoreCrc] <src> ...]
  79.     [-touchz <path> ...]
  80.     [-truncate [-w] <length> <path> ...]
  81.     [-usage [cmd ...]]
  83. Generic options supported are
  84. -conf <configuration file>     specify an application configuration file
  85. -D <property=value>            use value for given property
  86. -fs <local|namenode:port>      specify a namenode
  87. -jt <local|resourcemanager:port>    specify a ResourceManager
  88. -files <comma separated list of files>    specify comma separated files to be copied to the map reduce cluster
  89. -libjars <comma separated list of jars>    specify comma separated jar files to include in the classpath.
  90. -archives <comma separated list of archives>    specify comma separated archives to be unarchived on the compute machines.
  92. The general command line syntax is
  93. bin/hadoop command [genericOptions] [commandOptions]
  95. ----------
  97. ## 在HDFS上创建目录
  98. [hadoop@dev-161 hadoop]$ ./bin/hdfs dfs -mkdir -p /user/hadoop
  99. [hadoop@dev-161 hadoop]$ ./bin/hdfs dfs -mkdir input
  101. ## 查看HDFS的目录结构
  102. [hadoop@dev-161 hadoop]$ ./bin/hdfs dfs -ls /user/hadoop
  103. Found 1 items
  104. drwxr-xr-x   - hadoop supergroup          0 2017-08-23 14:40 /user/hadoop/input
  106. ## 上传本地文件至HDFS指定目录
  107. [hadoop@dev-161 hadoop]$ ./bin/hdfs dfs -put etc/hadoop/*.xml input
  109. [hadoop@dev-161 hadoop]$ ./bin/hdfs dfs -ls input
  110. Found 8 items
  111. -rw-r--r--   1 hadoop supergroup       4436 2017-08-23 14:40 input/capacity-scheduler.xml
  112. -rw-r--r--   1 hadoop supergroup       1115 2017-08-23 14:40 input/core-site.xml
  113. -rw-r--r--   1 hadoop supergroup       9683 2017-08-23 14:40 input/hadoop-policy.xml
  114. -rw-r--r--   1 hadoop supergroup       1180 2017-08-23 14:40 input/hdfs-site.xml
  115. -rw-r--r--   1 hadoop supergroup        620 2017-08-23 14:40 input/httpfs-site.xml
  116. -rw-r--r--   1 hadoop supergroup       3518 2017-08-23 14:40 input/kms-acls.xml
  117. -rw-r--r--   1 hadoop supergroup       5540 2017-08-23 14:40 input/kms-site.xml
  118. -rw-r--r--   1 hadoop supergroup        690 2017-08-23 14:40 input/yarn-site.xml
  120. [hadoop@dev-161 hadoop]$ ./bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.4.jar grep input output 'dfs[a-z.]+'
  122. ## 查看HDFS文件内容
  123. [hadoop@dev-161 hadoop]$ ./bin/hdfs dfs -cat output/*
  124. 1   dfsadmin
  125. 1   dfs.replication
  126. 1   dfs.namenode.name.dir
  127. 1   dfs.datanode.data.dir
  129. [hadoop@dev-161 hadoop]$ ./bin/hdfs dfs -ls
  130. Found 2 items
  131. drwxr-xr-x   - hadoop supergroup          0 2017-08-23 14:40 input
  132. drwxr-xr-x   - hadoop supergroup          0 2017-08-23 14:42 output
  134. ## 下载HDFS目录至本地
  135. [hadoop@dev-161 hadoop]$ ./bin/hdfs dfs -get output output
  136. [hadoop@dev-161 hadoop]$ cat output/*
  137. 1   dfsadmin
  138. 1   dfs.replication
  139. 1   dfs.namenode.name.dir
  140. 1   dfs.datanode.data.dir
  142. ## 删除HDFS指定目录
  143. [hadoop@dev-161 hadoop]$ ./bin/hdfs dfs -rm -r output
  144. 17/08/23 14:48:55 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
  145. Deleted output
  147. [hadoop@dev-161 hadoop]$ ./bin/hdfs dfs -ls
  148. Found 1 items
  149. drwxr-xr-x   - hadoop supergroup          0 2017-08-23 14:40 input
修改配置并启动YARN
  1. [hadoop@dev-161 hadoop]$ mv etc/hadoop/mapred-site.xml.template etc/hadoop/mapred-site.xml
  2. [hadoop@dev-161 hadoop]$ vim etc/hadoop/mapred-site.xml
  3. <configuration>
  4.         <property>
  5.              <name>mapreduce.framework.name</name>
  6.              <value>yarn</value>
  7.         </property>
  8. </configuration>
  10. [hadoop@dev-161 hadoop]$ vim etc/hadoop/yarn-site.xml
  11. <configuration>
  12.         <property>
  13.              <name>yarn.nodemanager.aux-services</name>
  14.              <value>mapreduce_shuffle</value>
  15.             </property>
  16. </configuration>
  18. ## 然后就可以启动 YARN 了（需要先执行过 ./sbin/start-dfs.sh）
  20. [hadoop@dev-161 hadoop]$ ./sbin/start-yarn.sh
  21. starting yarn daemons
  22. starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-resourcemanager-dev-161.out
  23. localhost: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-nodemanager-dev-161.out
  25. [hadoop@dev-161 hadoop]$ ./sbin/mr-jobhistory-daemon.sh start historyserver
  26. starting historyserver, logging to /usr/local/hadoop/logs/mapred-hadoop-historyserver-dev-161.out
  28. [hadoop@dev-161 hadoop]$ jps
  29. 5714 DataNode
  30. 5589 NameNode
  31. 7189 Jps
  32. 6726 ResourceManager
  33. 6838 NodeManager
  34. 5881 SecondaryNameNode
  35. 7150 JobHistoryServer
  37. ## webui
  38. http://172.16.1.161:8088/cluster

image_1bo6u2rd115jqsq413oc1nmt17ltm.png-150.9kB

添加新批注
在作者公开此批注前，只有你和作者可见。
回复批注