Kafka cluster deployment
System configuration adjustment
turn off the firewall
sudo systemctl stop ufw
sudo systemctl disable ufwResource allocation adjustment
edit/etc/security/limits.conffile
sudo vi /etc/security/limits.confAdd the following:
* soft nofile 65536
* hard nofile 65536
* soft nproc 65536
* hard nproc 65536Install and configure Kafka cluster
download and extract
cd /export/software
wget -P /export/software/ https://repo.huaweicloud.com/apache/kafka/3.4.0/kafka_2.12-3.4.0.tgz
tar -zxvf /export/software/kafka_2.12-3.4.0.tgz -C /export/servers/
cd /export/servers
mv /export/servers/kafka_2.12-3.4.0 /export/servers/kafkaConfigure Kafka server parameters
editorial changes/export/servers/kafka/config/server.propertiesfile
sudo chown -R hadooper:hadooper /export/servers/kafka/
vi /export/servers/kafka/config/server.propertieshadoop01：
broker.id=1
listeners=PLAINTEXT://hadoop01:9092
advertised.listeners=PLAINTEXT://hadoop01:9092
log.dirs=/export/servers/kafka/kafka-logs
num.partitions=3
zookeeper.connect=hadoop01:2181,hadoop02:2181,hadoop03:2181
default.replication.factor=3
offsets.topic.replication.factor=3vi /export/servers/kafka/bin/kafka-server-start.shAdd the following at the beginning of the script:
export JAVA_HOME=/export/servers/jdkvi /export/servers/kafka/bin/kafka-server-stop.shAdd the following at the beginning of the script:
export JAVA_HOME=/export/servers/jdkDelete and replace the original kafka in hive:
rm /export/servers/hive-4.0.1/lib/kafka-clients-2.5.0.jar
cp /export/servers/kafka/libs/kafka-clients-3.4.0.jar /export/servers/hive-4.0.1/lib/Distributed to hadoop02 and hadoop03 nodes at hadoop01 node
scp -r /export/servers/kafka hadoop02:/tmp/
scp -r /export/servers/kafka hadoop03:/tmp/Enter hadoop02:
sudo cp -r /tmp/kafka /export/servers/
rm -r /tmp/kafkaeditorial changes/export/servers/kafka/config/server.propertiesfile
sudo chown -R hadooper:hadooper /export/servers/kafka/
vi /export/servers/kafka/config/server.propertieshadoop02:
broker.id=2
listeners=PLAINTEXT://hadoop02:9092
advertised.listeners=PLAINTEXT://hadoop02:9092
log.dirs=/export/servers/kafka/kafka-logs
num.partitions=3
zookeeper.connect=hadoop01:2181,hadoop02:2181,hadoop03:2181
default.replication.factor=3
offsets.topic.replication.factor=3Enter hadoop03:
sudo cp -r /tmp/kafka /export/servers/
rm -r /tmp/kafkaeditorial changes/export/servers/kafka/config/server.propertiesfile
sudo chown -R hadooper:hadooper /export/servers/kafka/
vi /export/servers/kafka/config/server.propertieshadoop03:
broker.id=3
listeners=PLAINTEXT://hadoop03:9092
advertised.listeners=PLAINTEXT://hadoop03:9092
log.dirs=/export/servers/kafka/kafka-logs
num.partitions=3
zookeeper.connect=hadoop01:2181,hadoop02:2181,hadoop03:2181
default.replication.factor=3
offsets.topic.replication.factor=3Start cluster verification
Start the ZooKeeper cluster: Start the ZooKeeper service on each node
initiating node
zkServer.sh startCheck node role (Leader/Follower)
zkServer.sh statusLaunch Kafka cluster
On each Kafka node in order, start the Kafka service using the following command:
cd /export/servers/kafka
mkdir -p /export/servers/kafka/logs
mkdir -p /export/servers/kafka/kafka-logs
/export/servers/kafka/bin/kafka-server-start.sh -daemon /export/servers/kafka/config/server.propertiestest
jpsIf Quorum PeerMain and Kafka appear at the same time, the service was successfully launched, as shown in the following figure.
Test cluster working status
Create a topic on hadoop01:
/export/servers/kafka/bin/kafka-topics.sh --create --bootstrap-server hadoop01:9092,hadoop02:9092,hadoop03:9092 --replication-factor 3 --partitions 3 --topic my-test-topicCheck the topic description to confirm partition and replica allocation
/export/servers/kafka/bin/kafka-topics.sh --describe --bootstrap-server hadoop01:9092 --topic my-test-topicVerify whether the production and consumer terminals communicate with each other
hadoop01(producer):
/export/servers/kafka/bin/kafka-console-producer.sh --bootstrap-server hadoop01:9092 --topic my-test-topichadoop02(consumer):
/export/servers/kafka/bin/kafka-console-consumer.sh --bootstrap-server hadoop02:9092 --topic my-test-topic --from-beginninghadoo03(consumer):
/export/servers/kafka/bin/kafka-console-consumer.sh --bootstrap-server hadoop03:9092 --topic my-test-topic --from-beginningTest cluster status
Enter data in hadoop01 and view data in hadoop02 and hadoop03. If all of them are interconnected, it means that the kafka cluster is operating normally, as shown in the following figure
If you enjoyed this, leave a comment~