Kafka cluster deployment

System configuration adjustment

turn off the firewall

sudo systemctl stop ufw
sudo systemctl disable ufw

Resource allocation adjustment

Editing /etc/security/limits.conf files

sudo vi /etc/security/limits.conf

Add the following:

* soft nofile 65536
* hard nofile 65536
* soft nproc 65536
* hard nproc 65536

Install and configure Kafka cluster

download and extract

cd /export/software
wget -P /export/software/ https://repo.huaweicloud.com/apache/kafka/3.4.0/kafka_2.12-3.4.0.tgz
tar -zxvf /export/software/kafka_2.12-3.4.0.tgz -C /export/servers/
cd /export/servers
mv /export/servers/kafka_2.12-3.4.0 /export/servers/kafka

Configure Kafka server parameters

Edit and modify /export/servers/kafka/config/server.properties file

sudo chown -R hadooper:hadooper /export/servers/kafka/
vi /export/servers/kafka/config/server.properties

hadoop01：

broker.id=1
listeners=PLAINTEXT://hadoop01:9092
advertised.listeners=PLAINTEXT://hadoop01:9092
log.dirs=/export/servers/kafka/kafka-logs
num.partitions=3
zookeeper.connect=hadoop01:2181,hadoop02:2181,hadoop03:2181
default.replication.factor=3
offsets.topic.replication.factor=3

vi /export/servers/kafka/bin/kafka-server-start.sh

Add the following at the beginning of the script:

export JAVA_HOME=/export/servers/jdk

vi /export/servers/kafka/bin/kafka-server-stop.sh

Add the following at the beginning of the script:

export JAVA_HOME=/export/servers/jdk

Delete and replace the original kafka in hive:

rm /export/servers/hive-4.0.1/lib/kafka-clients-2.5.0.jar
cp /export/servers/kafka/libs/kafka-clients-3.4.0.jar /export/servers/hive-4.0.1/lib/

Distributed to hadoop02 and hadoop03 nodes at hadoop01 node

scp -r /export/servers/kafka hadoop02:/tmp/
scp -r /export/servers/kafka hadoop03:/tmp/

Enter hadoop02:

sudo cp -r /tmp/kafka /export/servers/
rm -r /tmp/kafka

Edit and modify /export/servers/kafka/config/server.properties file

sudo chown -R hadooper:hadooper /export/servers/kafka/
vi /export/servers/kafka/config/server.properties

hadoop02:

broker.id=2
listeners=PLAINTEXT://hadoop02:9092
advertised.listeners=PLAINTEXT://hadoop02:9092
log.dirs=/export/servers/kafka/kafka-logs
num.partitions=3
zookeeper.connect=hadoop01:2181,hadoop02:2181,hadoop03:2181
default.replication.factor=3
offsets.topic.replication.factor=3

Enter hadoop03:

sudo cp -r /tmp/kafka /export/servers/
rm -r /tmp/kafka

Edit and modify /export/servers/kafka/config/server.properties file

sudo chown -R hadooper:hadooper /export/servers/kafka/
vi /export/servers/kafka/config/server.properties

hadoop03:

broker.id=3
listeners=PLAINTEXT://hadoop03:9092
advertised.listeners=PLAINTEXT://hadoop03:9092
log.dirs=/export/servers/kafka/kafka-logs
num.partitions=3
zookeeper.connect=hadoop01:2181,hadoop02:2181,hadoop03:2181
default.replication.factor=3
offsets.topic.replication.factor=3

Start cluster verification

Start the ZooKeeper cluster: Start the ZooKeeper service on each node

initiating node

zkServer.sh start

Check node role (Leader/Follower)

zkServer.sh status

Launch Kafka cluster

On each Kafka node in order, start the Kafka service using the following command:

cd /export/servers/kafka
mkdir -p /export/servers/kafka/logs
mkdir -p /export/servers/kafka/kafka-logs
/export/servers/kafka/bin/kafka-server-start.sh -daemon /export/servers/kafka/config/server.properties

test

jps

If Quorum PeerMain and Kafka appear at the same time, the service was successfully launched, as shown in the following figure.

Test cluster working status

Create a topic on hadoop01:

/export/servers/kafka/bin/kafka-topics.sh --create --bootstrap-server hadoop01:9092,hadoop02:9092,hadoop03:9092 --replication-factor 3 --partitions 3 --topic my-test-topic

Check the topic description to confirm partition and replica allocation

/export/servers/kafka/bin/kafka-topics.sh --describe --bootstrap-server hadoop01:9092 --topic my-test-topic

Verify whether the production and consumer terminals communicate with each other

hadoop01(producer):

/export/servers/kafka/bin/kafka-console-producer.sh --bootstrap-server hadoop01:9092 --topic my-test-topic

hadoop02(consumer):

/export/servers/kafka/bin/kafka-console-consumer.sh --bootstrap-server hadoop02:9092 --topic my-test-topic --from-beginning

hadoo03(consumer):

/export/servers/kafka/bin/kafka-console-consumer.sh --bootstrap-server hadoop03:9092 --topic my-test-topic --from-beginning

Test cluster status

Enter data in hadoop01 and view data in hadoop02 and hadoop03. If all of them are interconnected, it means that the kafka cluster is operating normally, as shown in the following figure