Flink deployment documentation

Note: Flink deployment requires completion and launch of hadoop cluster and zookeeper deployment in advance

Environmental preparation

Download the Flink installation package

cd /export/software

wget -P /export/software/ https://mirrors.huaweicloud.com/apache/flink/flink-1.18.0/flink-1.18.0-bin-scala_2.12.tgz

Unpack Flink

tar -zxvf flink-1.18.0-bin-scala_2.12.tgz -C /export/servers/

cd /export/servers

mv flink-1.18.0 flink

Configure environment variables

sudo vim /etc/profile

Add at the end of the document:

#Flink环境变量
export FLINK_HOME=/export/servers/flink 
export PATH=$PATH:$FLINK_HOME/bin

Refresh environment variables

source /etc/profile

Configure Flink

Configure Hadoop environment

cd $FLINK_HOME

mkdir -p conf/hadoop 

cp /export/servers/hadoop/etc/hadoop/core-site.xml conf/hadoop/ 

cp /export/servers/hadoop/etc/hadoop/hdfs-site.xml conf/hadoop/

Configure flink-conf.yaml

vim conf/flink-conf.yaml

Add at the end of the file (copying and pasting will add a layer of #to the original content, and be careful to delete unnecessary #):

# 1.指定高可用模式为 ZooKeeper (利用现有的 zk 集群)
high-availability: zookeeper

# 2. 指定 ZooKeeper 集群地址 (参考文档中 ZooKeeper 章节的配置)
high-availability.zookeeper.quorum: hadoop01:2181,hadoop02:2181,hadoop03:2181

# 3. Flink 元数据在 HDFS 上的存储路径 (注意: mycluster 是你的 HDFS 逻辑名称)
high-availability.storageDir: hdfs://mycluster/flink/ha/

# 4. JobManager 的内存 (根据你的机器配置调整，文档中建议最低 4G 内存)
jobmanager.memory.process.size: 2048m

# 5. TaskManager 的内存和槽位
taskmanager.memory.process.size: 4096m
taskmanager.numberOfTaskSlots: 4

# 6. 指定 Hadoop 配置路径 (指向上面复制的文件夹)
env.java.opts: "-Dhadoop.user.group.static.mapping.overrides=hadooper:hadooper"

Create a directory on HDFS

Note: Start the Hadoop cluster according to the startup method in the document and then enter the following instructions

hdfs dfs -mkdir -p /flink/ha

hdfs dfs -chown -R hadooper:hadooper /flink

Launch and submit tasks

Mode 1 (Per-Job Mode)

Start Flink Session (on YARN)

cd $FLINK_HOME

./bin/yarn-session.sh -d -jm 1024 -tm 4096 -s 4 -nm flink-on-yarn-session

submit a job

./bin/flink run ./examples/streaming/WordCount.jar

Mode 2 (Application Mode)

Submit jobs directly to YARN without having to start a Session beforehand

./bin/flink run-application -t yarn-application ./examples/streaming/WordCount.jar

verification

View YARN Web UI

Open the browser and visit http://(IP of hadoop01)

View Flink Web UI

This UI cannot be opened when all tasks are completed, as long as the display in YARN is normal.

Enter the IP of hadoop01 in the browser and the port number displayed in the console to access For example: 192.168.1.1