oozie

Oozie workflow中四个Action解析--Sqoop Action

Sqoop Action部署

创建文件并写入表的数据

hadoop$:touch my_user.txt

创建sqoop在HDFS上的目录

hadoop$:bin/hdfs dfs -mkdir oozie-datas/sqoop
    hadoop$: bin/hdfs dfs -put my_user.txt oozie-datas/sqoop

拷贝example的sqoop项目

oozie$:cp -r examples/apps/sqoop oozie-apps/

在项目根目录下创建lib目录并拷贝mysqljar包

oozie$:mkdir oozie-apps/sqoop/lib
    oozie$:cp /opt/cdh3.5.6/hive-0.13.1-cdh5.3.6/lib/mysql-connector-java-5.1.27-bin.jar oozie-apps/sqoop/lib/

配置文件

• job.properties

nameNode=hdfs://hadoop-senior01.grc.com:8020
jobTracker=hadoop-senior01.grc.com:8032
queueName=default
oozieAppsRoot=user/grc/oozie-apps
oozieDataRoot=user/grc/oozie-datas

oozie.use.system.libpath=true
oozie.wf.application.path=${nameNode}/${oozieAppsRoot}/sqoop

• workflow.xml

<workflow-app xmlns="uri:oozie:workflow:0.5" name="sqoop-wf">
<start to="sqoop-node"/>
<action name="sqoop-node">
    <sqoop xmlns="uri:oozie:sqoop-action:0.3">
        <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
       <configuration>
            <property>
                <name>mapred.job.queue.name</name>
                <value>${queueName}</value>
            </property>
        </configuration>
        <command>export --connect jdbc:mysql://hadoop-senior01.grc.com:3306/db_1206 --username root --password 123456 --table my_user --num-mappers 1 --input-fields-terminated-by "\t" --export-dir /user/grc/oozie-datas/sqoop</command>
    </sqoop>
    <ok to="end"/>
    <error to="fail"/>
</action>

<kill name="fail">
    <message>Sqoop failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
</workflow-app>




/opt/cdh3.5.6/hadoop-2.5.0-cdh5.3.6/bin/hdfs dfs -put oozie-apps/sqoop/ oozie-apps

workflow.xml第二种配置

• workflow.xml

<workflow-app xmlns="uri:oozie:workflow:0.5" name="sqoop-wf">
<start to="sqoop-node"/>

<action name="sqoop-node">
    <sqoop xmlns="uri:oozie:sqoop-action:0.3">
        <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
       <configuration>
            <property>
                <name>mapred.job.queue.name</name>
                <value>${queueName}</value>
            </property>
        </configuration>
        <arg>export</arg>
        <arg>--connect</arg>
        <arg>jdbc:mysql://hadoop-senior01.grc.com:3306/db_1206</arg>
        <arg>--username</arg>
        <arg>root</arg>
        <arg>--password</arg>
        <arg>123456</arg>
        <arg>--table</arg>
        <arg>my_user</arg>
        <arg>--num-mappers</arg>
        <arg>1</arg>
        <arg>--input-fields-terminated-by</arg>
        <arg>"\t"</arg>
        <arg>--export-dir</arg>
        <arg>/user/grc/oozie-datas/sqoop</arg>
    </sqoop>
    <ok to="end"/>
    <error to="fail"/>
</action>

<kill name="fail">
    <message>Sqoop failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
</workflow-app>

Note:之间不能有空格，"\t"之间可以加双引号，或不加，但是不能加单引号。

参考地址：http://archive.cloudera.com/cdh5/cdh/5/oozie-4.0.0-cdh5.3.6/DG_SqoopActionExtension.html