oozie

Oozie工作流调度框架

开源的资源调度框架

• linux crontab 针对每个用户

• Azkaban 基本的调度框架，配置简单上手快，

• Oozie

• Zeus阿里的调度框架!
  

Oozie三大功能

Oozie安装部署

配置hadoop下的文件

配置core-site.xml

<!-- OOZIE -->
    <property>
        <name>hadoop.proxyuser.grc.hosts</name>
        <value>hadoop-senior01.grc.com</value>
    </property>
    <property>
        <name>hadoop.proxyuser.grc.groups</name>
        <value>*</value>
    </property>

解压软件

> tar -zxf oozie-hadooplibs-4.0.0-cdh5.3.6.tar.gz -C /opt/cdh3.5.6/

创建目录

mkdir libext

copyjar到libext

cp -r hadooplibs/hadooplib-2.5.0-cdh5.3.6.oozie-4.0.0-cdh5.3.6/* libext/

copy ext-2.2.zip到libext文件夹下

cp -r /opt/softwares/ext-2.2.zip libext/

oozie准备工作

bin/oozie-setup.sh prepare-war 

    bin/oozie-setup.sh sharelib create -fs \
    hdfs://hadoop-senior01.grc.com:8020 \
     -locallib oozie-sharelib-4.0.0-cdh5.3.6-yarn.tar.gz

bin/ooziedb.sh create -sqlfile oozie.sql -run DB Connection

启动命令

bin/oozied.sh start

bin/oozie admin -oozie http://hadoop-senior01.grc.com:11000/oozie -status

oozie example

解压example

tar -zxf oozie-examples.tar.gz  

将解压好的example文件上传到HDFS系统之上

/opt/cdh3.5.6/hadoop-2.5.0-cdh5.3.6/bin/hdfs dfs -put examples examples

> 注意：如果上传之前HDFS文件系统中存在，先删除后上传。

运行mapreduce

• 修改配置文件job.properties

nameNode=hdfs://hadoop-senior01.grc.com:8020
jobTracker=hadoop-senior01.grc.com:8032

• 运行方式一

bin/oozie job -oozie http://hadoop-senior01.grc.com:11000/oozie -config examples/apps/map-reduce/job.properties -run

• 运行方式二

  • 设置环境变量

export OOZIE_URL=http://hadoop-senior01.grc.com:11000/oozie/

直接运行

bin/oozie job -config examples/apps/map-reduce/job.properties -run

运行产生job id为 0000000-151219080030057-oozie-grc-W

• 查看workflow job状态

bin/oozie job -oozie http://hadoop-senior01.grc.com:11000/oozie -info 0000000-151219080030057-oozie-grc-W

问题

 Error starting action [mr-node]. ErrorType [FAILED], ErrorCode [It should never happe
n], Message [File /user/grc/share/lib does not exist]
org.apache.oozie.action.ActionExecutorException: File /user/grc/share/lib does not exist

原因：没有读取HDFS文件上的文件系统，而是读取的是本地的文件

<property>
    <name>oozie.service.HadoopAccessorService.hadoop.configurations</name>
    <value>*=/opt/cdh3.5.6/hadoop-2.5.0-cdh5.3.6/etc/hadoop/</value>
    <description>
        Comma separated AUTHORITY=HADOOP_CONF_DIR, where AUTHORITY is the HOST:PORT of
        the Hadoop service (JobTracker, HDFS). The wildcard '*' configuration is
        used when there is no exact match for an authority. The HADOOP_CONF_DIR contains
        the relevant Hadoop *-site.xml files. If the path is relative is looked within
        the Oozie configuration directory; though the path can be absolute (i.e. to point
        to Hadoop client conf/ directories in the local filesystem.
    </description>
</property>

oozie重启

bin/oozied.sh stop 
bin/oozied.sh start

重新运行程序

bin/oozie job -oozie http://hadoop-senior01.grc.com:11000/oozie -config examples/apps/map-reduce/job.properties -run

export OOZIE_URL=http://hadoop-senior01.grc.com:11000/oozie/

运行

bin/oozie job  -config examples/apps/map-reduce/job.properties -run

查看

bin/oozie job  -info 0000000-151219080030057-oozie-grc-W