Setting up Kerberos authentication for Hadoop with Cloudera Manager

Setting up Kerberos authentication for Hadoop with Cloudera Manager involves multiple steps. Below is a step-by-step guide with examples to help you through the process. Please note that this guide assumes you have Cloudera Manager and a Hadoop cluster already installed.

Step 1: Install Kerberos Packages

Install Kerberos client packages on all nodes in your cluster. On CentOS, you can use the following commands:

        sudo yum install krb5-workstation krb5-libs

Step 2: Configure Kerberos Server

Edit the /etc/krb5.conf file to configure the Kerberos realm and KDC settings. Here's an example of a minimal krb5.conf:

[libdefaults]

    default_realm = EXAMPLE.COM

 

[realms]

    EXAMPLE.COM = {

        kdc = kdc.example.com

    }

Step 3: Create Kerberos Principals

Create Kerberos principals for each Hadoop service and user that will use Kerberos authentication. For example, to create a principal for the HDFS service:

sudo kadmin.local -q "addprinc -randkey hdfs/your-hostname@EXAMPLE.COM"

Step 4: Generate Keytabs

Generate a keytab for each service and user principal. For example, to generate a keytab for the HDFS service:

sudo kadmin.local -q "xst -k /etc/security/keytabs/hdfs.keytab hdfs/your-hostname@EXAMPLE.COM"

sudo chmod 400 /etc/security/keytabs/hdfs.keytab

Step 5: Configure Hadoop Services

Use Cloudera Manager to configure each Hadoop service to use Kerberos. In Cloudera Manager, navigate to the service's configuration and search for "Kerberos." Configure the following properties:

• Kerberos Principal: The service principal (e.g., hdfs/your-hostname@EXAMPLE.COM).
• Kerberos Keytab: The path to the generated keytab (e.g., /etc/security/keytabs/hdfs.keytab).

Step 6: Enable Kerberos Authentication in Cloudera Manager

In Cloudera Manager, go to "Administration" > "Security" > "Kerberos" and click on "Enable Kerberos." Follow the wizard to provide KDC information, realm name (EXAMPLE.COM), and administrative credentials.

Step 7: Restart Services

After enabling Kerberos, Cloudera Manager will prompt you to restart the services. Restart all the services to apply Kerberos authentication.

Step 8: Test Kerberos Authentication

Test the Kerberos setup by accessing Hadoop services. For example, use kinit to authenticate as a user principal:

        kinit your-user@EXAMPLE.COM

Then, access HDFS:

        hdfs dfs -ls /

If successful, you won't be prompted for a password.

Step 9: Update Client Configuration

Update the client configuration files (core-site.xml, hdfs-site.xml, etc.) to include Kerberos-related settings. For example, in core-site.xml:

<property>

    <name>hadoop.security.authentication</name>

    <value>kerberos</value>

</property>

<property>

    <name>hadoop.security.authorization</name>

    <value>true</value>

</property>

Step 10: Monitor and Maintain

Regularly monitor the Kerberos authentication setup. Renew keytabs as needed before they expire. Monitor the Cloudera Manager Kerberos health checks.

Remember that the process can vary depending on your specific setup and Cloudera Manager version. Always refer to Cloudera's official documentation for detailed instructions tailored to your environment.