hadoop

Hadoop2.5.0编译教程

1. 准备环境

*jdk:jdk-7u67-linux-x64.tar.gz
*hadoop:hadoop-2.5.0-src.tar.gz
*maven:apache-maven-3.0.5-bin.tar.gz
*protobuf:protobuf-2.5.0.tar.gz

2. 搭建环境

编译hadoop的环境要求可以在hadoop-src根目录下BUILDING.txt所示：
Build instructions for Hadoop

Requirements:

• Unix System
• JDK 1.6+
• Maven 3.0 or later
• Findbugs 1.3.9 (if running findbugs)
• ProtocolBuffer 2.5.0
• CMake 2.6 or newer (if compiling native code)
• Zlib devel (if compiling native code)
• openssl devel ( if compiling native hadoop-pipes )
• Internet connection for first build (to fetch all Maven and Hadoop dependencie
  s)

2.1 jdk安装及环境配置

jdk压缩包解压：

[grc@bigdata-senior01 softwares]$ tar -zxvf jdk-7u67-linux-x64.tar.gz -C /opt/mudules/

配置环境变量：

[grc@bigdata-senior01 softwares]$ sudo vim /etc/profile

##JAVA_HOME
export JAVA_HOME=/opt/mudules/jdk1.7.0_67
export PATH=$JAVA_HOME/bin:$PATH
[grc@bigdata-senior01 softwares]$ source /etc/profile

jdk验证：

[grc@bigdata-senior01 softwares]$ java -version
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

2.2 maven安装及环境配置

maevn压缩包解压：

[grc@bigdata-senior01 softwares]$ tar -zxvf apache-maven-3.0.5-bin.tar.gz -C /opt/mudules/

配置环境变量：

[grc@bigdata-senior01 softwares]$ vim /etc/profile
    ##MAEVN_HOME
    export MAEVN_HOME=/opt/mudules/apache-maven-3.0.5
    export PATH=$MAEVN_HOME/bin:$PATH

maevn验证：

[grc@bigdata-senior01 softwares]$ mvn -version
Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19 08:51:28-0500)
Maven home: /opt/mudules/apache-maven-3.0.5
Java version: 1.7.0_67, vendor: Oracle Corporation
Java home: /opt/mudules/jdk1.7.0_67/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-358.el6.x86_64", arch: "amd64", family: "unix"

2.3 protobuf安装及环境配置

在安装 protobuf前，需要安装gcc/gcc-c++/make

[grc@bigdata-senior01 ~]$ sudo yum install gcc
[grc@bigdata-senior01 ~]$ sudo yum install gcc-c++
[grc@bigdata-senior01 ~]$ sudo yum install make

protobuf压缩包解压：

[grc@bigdata-senior01 softwares]$ tar -zxvf protobuf-2.5.0.tar.gz -C /opt/mudules/
[grc@bigdata-senior01 protobuf-2.5.0]$ ./configure -prefix=/usr/local/protoc/
[grc@bigdata-senior01 protobuf-2.5.0]$ sudo make

配置环境变量：

[grc@bigdata-senior01 softwares]$ vim /etc/profile
    ##PROTOC_HOME
    export PATH=.:/usr/local/protoc/bin:$PATH

protocbuf验证：

[grc@bigdata-senior01 softwares]$ source /etc/profile
[grc@bigdata-senior01 protobuf-2.5.0]$ protoc --version
libprotoc 2.5.0

2.4 安装其他依赖包

[grc@bigdata-senior01 ]$ sudo yum install cmake
    [grc@bigdata-senior01 ]$ sudo yum install openssl-devel
[grc@bigdata-senior01 ]$ sudo yum install ncurses-devel

3.编译

[grc@bigdata-senior01 hadoop-2.5.0-src]$ mvn package -DskipTests -Pdist,native

编译结果：

[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop Main ................................ SUCCESS [2.835s]
[INFO] Apache Hadoop Project POM ......................... SUCCESS [2.019s]
[INFO] Apache Hadoop Annotations ......................... SUCCESS [3.619s]
[INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.329s]
[INFO] Apache Hadoop Project Dist POM .................... SUCCESS [1.912s]
[INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [4.263s]
[INFO] Apache Hadoop MiniKDC ............................. SUCCESS [3.951s]
[INFO] Apache Hadoop Auth ................................ SUCCESS [4.970s]
[INFO] Apache Hadoop Auth Examples ....................... SUCCESS [3.286s]
[INFO] Apache Hadoop Common .............................. SUCCESS [1:54.184s]
[INFO] Apache Hadoop NFS ................................. SUCCESS [13.824s]
[INFO] Apache Hadoop Common Project ...................... SUCCESS [0.095s]
[INFO] Apache Hadoop HDFS ................................ SUCCESS [4:48.429s]
[INFO] Apache Hadoop HttpFS .............................. SUCCESS [32.920s]
[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [13.930s]
[INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [5.379s]
[INFO] Apache Hadoop HDFS Project ........................ SUCCESS [0.523s]
[INFO] hadoop-yarn ....................................... SUCCESS [0.074s]
[INFO] hadoop-yarn-api ................................... SUCCESS [1:13.810s]
[INFO] hadoop-yarn-common ................................ SUCCESS [39.062s]
[INFO] hadoop-yarn-server ................................ SUCCESS [0.160s]
[INFO] hadoop-yarn-server-common ......................... SUCCESS [17.656s]
[INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [19.529s]
[INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [3.686s]
[INFO] hadoop-yarn-server-applicationhistoryservice ...... SUCCESS [7.730s]
[INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [18.782s]
[INFO] hadoop-yarn-server-tests .......................... SUCCESS [1.078s]
[INFO] hadoop-yarn-client ................................ SUCCESS [7.807s]
[INFO] hadoop-yarn-applications .......................... SUCCESS [0.059s]
[INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [2.680s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [1.999s]
[INFO] hadoop-yarn-site .................................. SUCCESS [0.158s]
[INFO] hadoop-yarn-project ............................... SUCCESS [5.662s]
[INFO] hadoop-mapreduce-client ........................... SUCCESS [0.132s]
[INFO] hadoop-mapreduce-client-core ...................... SUCCESS [28.540s]
[INFO] hadoop-mapreduce-client-common .................... SUCCESS [26.310s]
[INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [7.969s]
[INFO] hadoop-mapreduce-client-app ....................... SUCCESS [12.131s]
[INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [11.498s]
[INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [5.794s]
[INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [2.244s]
[INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [7.783s]
[INFO] hadoop-mapreduce .................................. SUCCESS [4.799s]
[INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [6.196s]
[INFO] Apache Hadoop Distributed Copy .................... SUCCESS [13.284s]
[INFO] Apache Hadoop Archives ............................ SUCCESS [4.517s]
[INFO] Apache Hadoop Rumen ............................... SUCCESS [8.540s]
[INFO] Apache Hadoop Gridmix ............................. SUCCESS [5.399s]
[INFO] Apache Hadoop Data Join ........................... SUCCESS [3.134s]
[INFO] Apache Hadoop Extras .............................. SUCCESS [3.451s]
[INFO] Apache Hadoop Pipes ............................... SUCCESS [4.132s]
[INFO] Apache Hadoop OpenStack support ................... SUCCESS [7.975s]
[INFO] Apache Hadoop Client .............................. SUCCESS [6.242s]
[INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [0.203s]
[INFO] Apache Hadoop Scheduler Load Simulator ............ SUCCESS [7.672s]
[INFO] Apache Hadoop Tools Dist .......................... SUCCESS [4.450s]
[INFO] Apache Hadoop Tools ............................... SUCCESS [0.062s]
[INFO] Apache Hadoop Distribution ........................ SUCCESS [33.432s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 15:16.557s
[INFO] Finished at: Wed Nov 11 15:48:47 EST 2015
[INFO] Final Memory: 76M/241M
[INFO] ------------------------------------------------------------------------

注意：编译后的代码在hadoop-2.5.0-src/hadoop-dist/target/hadoop-2.5.0下。