Installing Apache Kafka with 3 Nodes on CentOS

 

 

Installing Apache Kafka with 3 Nodes on CentOS

Apache Kafka is a popular distributed streaming platform that is widely used for building real-time data pipelines and streaming applications. In this tutorial, we'll guide you through the process of setting up a Kafka cluster with three nodes on CentOS.

Prerequisites

Before we start, ensure that you have the following prerequisites:

1. Three CentOS machines (nodes) with Java (OpenJDK) installed.
2. SSH access to each node.

Step 1: Download and Extract Kafka

On each node, download and extract the Kafka binaries:

bash

wget https://downloads.apache.org/kafka/3.0.0/kafka_2.13-3.0.0.tgz

tar -xzf kafka_2.13-3.0.0.tgz

cd kafka_2.13-3.0.0

Step 2: Configure ZooKeeper

Edit the config/server.properties file on each node to configure ZooKeeper and Kafka broker properties:

bash

                    nano config/server.properties

Update the following properties:

properties

broker.id=<unique-id-for-each-node>

listeners=PLAINTEXT://<node-ip>:9092

advertised.listeners=PLAINTEXT://<node-ip>:9092

Replace <unique-id-for-each-node> with a unique identifier for each broker, and <node-ip> with the IP address of the corresponding node.

Step 3: Create ZooKeeper myid File

Create a myid file for ZooKeeper on each node:

bash

                    echo "<unique-id-for-each-node>" > /tmp/zookeeper/myid

Replace <unique-id-for-each-node> with the same unique identifier used for Kafka broker id.

Step 4: Start ZooKeeper

On each node, start ZooKeeper:

bash

                    bin/zookeeper-server-start.sh config/zookeeper.properties

Step 5: Start Kafka

On each node, start Kafka:

bash

                    bin/kafka-server-start.sh config/server.properties

Step 6: Create Topics (Optional)

You can create topics on any node:

bash

    bin/kafka-topics.sh --create --topic test-topic --bootstrap-server <node-ip>:9092 --partitions 3 --replication-factor 3

Adjust <node-ip> accordingly.

Step 7: Verify the Cluster

Check the status of the Kafka brokers:

bash

                    bin/kafka-topics.sh --list --bootstrap-server <node-ip>:9092

Step 8: Produce and Consume Messages

Test producing and consuming messages using the Kafka console scripts.

Congratulations! You have successfully set up a Kafka cluster with three nodes on CentOS.