Apache Oozie 3.3.1 installation on Apache Hadoop 0.23.0

Hi
I have been trying to install apache oozie 3.3.1 on Hadoop 0.23.0. from last few days.
The documentation provided in apache website is not very clear and very less documentation is provided for the hadoop with new (MRv2 / Yarn) architecture. So I hope my blog will help to some extent in configuring oozie 3.3.1 on Hadoop 0.23.0

Here we go,

link to apache oozie quick start http://oozie.apache.org/docs/3.3.1/DG_QuickStart.html
and apache hadoop http://hadoop.apache.org/docs/r0.23.0
My testing environment

4 node cluster ( 1 master , 3slaves )

Apache Hadoop 0.23.0

Apache oozie 3.3.1

java 1.6.0_26

Maven 3.0.4

Oozie server installation

Download oozie 3.3.1.tar.gz from the nearest mirror site apache/oozie/3.3.1 ( i downloaded from nus.edu.sg mirror)
Unpack the oozie-3.3.1 tar.gz file under some /home/srikanth
The following two properties are required in Hadoop core-site.xml:
```
  
  <property>
    <name>hadoop.proxyuser.[OOZIE_SERVER_USER].hosts</name>
    <value>[OOZIE_SERVER_HOSTNAME]</value>
  </property>
  <property>
    <name>hadoop.proxyuser.[OOZIE_SERVER_USER].groups</name>
    <value>[USER_GROUPS_THAT_ALLOW_IMPERSONATION]</value>
  </property>
```
[ I replaced [OOZIE_SERVER_USER] with my srikanth and [OOZIE_SERVER_HOSTNAME] with * [USER_GROUPS_THAT_ALLOW_IMPERSONATION] with *
Replace the capital letter sections with specific values and then restart Hadoop.
(For this blog purpose is set /home/srikanth/oozie-3.3.1 as my $OOZIE_HOME)
Download ExtJS (e.g. extjs-2.2.zip). Do not unzip it right away. At this point, the instructions in the Quick Start Guide ( a.k.a. QSG) jump to calling bin/setup.sh, which will not work. Also, the path to setup.sh is incorrectly specified in the QSG.
Building a Distro

> cd $OOZIE_HOME
   > ./bin/mkdistro.sh -DskipTests
       If successful, you will see the following print out:

Oozie distro created, DATE[2013.01.02-01:38:05GMT] VC-REV[unavailable], available at [/home/srikanth/oozie-3.3.1/distro/target]

In $OOZIE_HOME a directory called *hadooplibs/* directory will be created containing the Hadoop JARs for the versions of Hadoop that the Oozie distribution supports.
Create a libext/ directory in the directory where Oozie was expanded.
If using a version of Hadoop bundled in Oozie hadooplibs/ , copy the corresponding Hadoop JARs from hadooplibs/ to the libext/ directory.
If using a different version of Hadoop, copy the required Hadoop JARs from such version in the libext/ directory. ( since hadoop 0.23.0 is not in the versions given in hadooplibs/, i copied the required jars from hadoop-0.23.0 to libext/. )

Note : I did the following steps to copy the hadoop jars into libext/ directory. Hope this will give some idea what jars to select and copy
                 > cd $HADOOP_HOME/lib
                     > cp *.jar ../../oozie-3.3.1/libext/
                     > cd ..
                     > cp *.jar ../../oozie-3.3.1/libext/
                     > cp hadoop-mapreduce-tools-0.23.0.jar hadoop-mapreduce-tools-0.23.0-sources.jar hadoop-mapreduce-test-0.23.0.jar hadoop-mapreduce-test-0.23.0-sources.jar hadoop-mapreduce-examples-0.23.0.jar hadoop-mapreduce-examples-0.23.0-sources.jar hadoop-mapreduce-0.23.0.jar hadoop-mapreduce-0.23.0-sources.jar hadoop-0.23.0-streaming.jar hadoop-0.23.0-gridmix.jar /home/srikmvm/oozie-3.3.1/libext/

Building with ExtJS library for Oozie Web Console

Copy the extjs-2.2.zip over to $OOZIE_HOME/webapp/src/main/webapp/
Unzip it

Run oozie-setup.sh to create an oozie.war file

> cd $OOZIE_HOME
> ./distro/target/oozie-3.3.1-distro/oozie-3.3.1/bin/oozie-setup.sh -hadoop 0.23.0 /home/srikanth/hadoop-0.23.0 -extjs $OOZIE_HOME/webapp/src/main/webapp/ext-2.2.zip
if the command is successfully run ,
New Oozie WAR file will be created by injecting ‘Hadoop JARs, ExtJS library’ at /home/srikanth/oozie-3.3.1/distro/target/oozie-3.3.1-distro/oozie-3.3.1/oozie-server/webapps/oozie.war
INFO: Oozie is ready to be started

Copy the newly minted oozie.war file to your Tomcat deployment directory
> cd $OOZIE_HOME
> cp ./distro/target/oozie-3.3.1-distro/oozie-3.3.1/oozie-server/webapps/oozie.war ./webapp/src/main/webapp/oozie.war

Make a configuration change before starting the Oozie Tomcat App

> cd $OOZIE_HOME
> vi ./distro/target/oozie-3.3.1-distro/oozie-3.3.1/conf/oozie-site.xml Change the property below to true as shown. It is originally false, which causes startup to fail.

<property>

        <name>oozie.service.JPAService.create.db.schema</name>

<value>true</value>

        <description>

            Creates Oozie DB.

If set to true, it creates the DB schema if it does not exist. If the DB schema exists is a NOP.

            If set to false, it does not create the DB schema. If the DB schema does not exist it fails start up.

        </description>

    </property>

Run the following command

> cd $OOZIE_HOME

> ./distro/target/oozie-3.3.1-distro/oozie-3.3.1/bin/ooziedb.sh create -sqlfile oozie.sql -run

if the command is successfully run, we can the following

          DONE
          Check DB schema does not exist
          DONE
          Check OOZIE_SYS table does not exist
          DONE
          Create SQL schema
          DONE
          DONE
          Create OOZIE_SYS table
          DONE
          Oozie DB has been created for Oozie version '3.3.1'

```
 Start the Oozie Server
```
> cd $OOZIE_HOME
> ./distro/target/oozie-3.3.1-distro/oozie-3.3.1/bin/oozie-start.sh
```
 
```

To start Oozie as a foreground process run:

> cd $OOZIE_HOME

> ./distro/target/oozie-3.3.1-distro/oozie-3.3.1/bin/oozie-run.sh
```
 
```
Check the Oozie log file ./distro/target/oozie-3.3.1-distro/oozie-3.3.1/logs/oozie.log to ensure Oozie started properly.

Using the Oozie command line tool check the status of Oozie:
$ ./distro/target/oozie-3.3.1-distro/oozie-3.3.1/bin/oozie admin -oozie http://[ip address of namenode]:11000/oozie -status

Using a browser go to the Oozie web console, Oozie status should be NORMAL

Tech Insights

About Us

Contact Form