oozie
Oozie调度框架

开源的资源调度框架

• linux crontab 针对每个用户

• Azkaban 基本的调度框架，配置简单上手快，

• Oozie

• Zeus阿里的调度框架!

Oozie三大功能

Oozie安装部署

配置hadoop下的文件

配置core-site.xml

<property>

    <name>hadoop.proxyuser.grc.hosts</name>

    <value>hadoop-senior01.grc.com</value>

</property>

<property>

    <name>hadoop.proxyuser.grc.groups</name>

    <value>*</value>

</property>

解压软件

> tar -zxf oozie-hadooplibs-4.0.0-cdh5.3.6.tar.gz -C /opt/cdh3.5.6/

创建目录

mkdir libext

解压文件

tar -zxf oozie-hadooplibs-4.0.0-cdh5.3.6.tar.gz

copyjar到libext

cp -r hadooplibs/hadooplib-2.5.0-cdh5.3.6.oozie-4.0.0-cdh5.3.6/* libext/

copy ext-2.2.zip到libext文件夹下

cp -r /opt/softwares/ext-2.2.zip libext/

oozie准备工作

bin/oozie-setup.sh prepare-war 

    bin/oozie-setup.sh sharelib create -fs \

    hdfs://hadoop-senior01.grc.com:8020 \

     -locallib oozie-sharelib-4.0.0-cdh5.3.6-yarn.tar.gz

bin/ooziedb.sh create -sqlfile oozie.sql -run DB Connection

启动命令

bin/oozied.sh start

bin/oozie admin -oozie http://hadoop-senior01.grc.com:11000/oozie -status

oozie example

解压example

tar -zxf oozie-examples.tar.gz  

将解压好的example文件上传到HDFS系统之上

/opt/cdh3.5.6/hadoop-2.5.0-cdh5.3.6/bin/hdfs dfs -put examples examples



> 注意：如果上传之前HDFS文件系统中存在，先删除后上传。

运行mapreduce

• 修改配置文件job.properties

nameNode=hdfs://hadoop-senior01.grc.com:8020

jobTracker=hadoop-senior01.grc.com:8032

• 运行方式一

bin/oozie job -oozie http://hadoop-senior01.grc.com:11000/oozie -config examples/apps/map-reduce/job.properties -run

• 运行方式二

  • 设置环境变量

export OOZIE_URL=http://hadoop-senior01.grc.com:11000/oozie/

直接运行

bin/oozie job -config examples/apps/map-reduce/job.properties -run

运行产生job id为 0000000-151219080030057-oozie-grc-W

• 查看workflow job状态

bin/oozie job -oozie http://hadoop-senior01.grc.com:11000/oozie -info 0000000-151219080030057-oozie-grc-W

问题

 Error starting action [mr-node]. ErrorType [FAILED], ErrorCode [It should never happe

n], Message [File /user/grc/share/lib does not exist]

org.apache.oozie.action.ActionExecutorException: File /user/grc/share/lib does not exist

原因：没有读取HDFS文件上的文件系统，而是读取的是本地的文件

<property>

    <name>oozie.service.HadoopAccessorService.hadoop.configurations</name>

    <value>*=/opt/cdh3.5.6/hadoop-2.5.0-cdh5.3.6/etc/hadoop/</value>

    <description>

        Comma separated AUTHORITY=HADOOP_CONF_DIR, where AUTHORITY is the HOST:PORT of

        the Hadoop service (JobTracker, HDFS). The wildcard '*' configuration is

        used when there is no exact match for an authority. The HADOOP_CONF_DIR contains

        the relevant Hadoop *-site.xml files. If the path is relative is looked within

        the Oozie configuration directory; though the path can be absolute (i.e. to point

        to Hadoop client conf/ directories in the local filesystem.

    </description>

</property>

oozie重启

bin/oozied.sh stop 

bin/oozied.sh start

重新运行程序

bin/oozie job -oozie http://hadoop-senior01.grc.com:11000/oozie -config examples/apps/map-reduce/job.properties -run

export OOZIE_URL=http://hadoop-senior01.grc.com:11000/oozie/

运行

bin/oozie job  -config examples/apps/map-reduce/job.properties -run

查看

bin/oozie job  -info 0000000-151219080030057-oozie-grc-W