Spark cluster deployment

Installation and configuration of Spark

Download the Spark installation package

wget -P /export/software/ https://repo.huaweicloud.com/apache/spark/spark-3.4.3/spark-3.4.3-bin-hadoop3.tgz

Unpack the Spark installation package

tar -zxvf /export/software/spark-3.4.3-bin-hadoop3.tgz -C /export/servers
# 重命名:
cd /export/servers
mv /export/servers/spark-3.4.3-bin-hadoop3 /export/servers/spark

modify the configuration file

Enter the spark/conf directory and modify Spark’s configuration file spark-env.sh. Copy a copy of the spark-env.sh.template configuration template file and name it spark-env.sh. The command is as follows:

cd /export/servers/spark/conf
sudo cp /export/servers/spark/conf/spark-env.sh.template /export/servers/spark/conf/spark-env.sh

Modify spark-env.sh file

sudo vim /export/servers/spark/conf/spark-env.sh

Add the following to the file:

#配置java环境变量
export JAVA_HOME=/export/servers/jdk
#指定Master的IP
export SPARK_MASTER_HOST=hadoop01
#指定Master的端口
export SPARK_MASTER_PORT=7077
export SPARK_MASTER_WEBUI_PORT=8082

Copy the workers.template file and rename it to workers

sudo cp /export/servers/spark/conf/workers.template /export/servers/spark/conf/workers

Edit workers configuration file

sudo vim /export/servers/spark/conf/workers

hadoop01
hadoop02
hadoop03

distribution of documents

scp -r /export/servers/spark/ hadoop02:/tmp/
scp -r /export/servers/spark/ hadoop03:/tmp/

Enter hadoop02:

sudo cp -r /tmp/spark /export/servers
rm -r /tmp/spark

Enter hadoop03:

sudo cp -r /tmp/spark /export/servers
rm -r /tmp/spark

Launch Spark cluster

Add permissions separately

cd /export/servers
sudo chown -R hadooper:hadooper /export/servers/spark/

Start Spark step by step

hadoop01:

cd /export/servers/spark/sbin/
/export/servers/spark/sbin/start-master.sh

hadoop02,hadoop03:

cd /export/servers/spark/sbin/
/export/servers/spark/sbin/start-slave.sh spark://hadoop01:7077

If there is a problem when starting hadoop02 and hadoop03 at this step, first check whether the host name is correct! If it is incorrect, modify it according to the tutorial on the Internet, and we will not go into it too much here.

Enter $SPARK_HOME/sbin/start-all.sh to start (the above steps are also starting, which is based on the server step by step, and this starting command is batch starting), which means success as shown below. Further verify that open the local browser and enter （hadoop01的IP地址）:8082. If the interface opens normally, it means success.

Adapting Spark to YARN

Configure Spark to recognize YARN

Go to the configuration directory /export/servers/spark/conf and modify the spark-env.sh file

cd /export/servers/spark/conf
sudo vim /export/servers/spark/conf/spark-env.sh

Add the following:

# 设置Hadoop配置文件目录（请根据你的实际路径修改）
export HADOOP_CONF_DIR=/export/servers/hadoop/etc/hadoop

Comment out the following (comment with #):

export SPARK_MASTER_HOST=hadoop01
export SPARK_MASTER_PORT=7077
export SPARK_MASTER_WEBUI_PORT=8082

Adjust YARN configuration

Enter the configuration directory

cd /export/servers/hadoop/etc/hadoop

Modify yarn-site.xml file

sudo vim /export/servers/hadoop/etc/hadoop/yarn-site.xml

Add the following content (note that the indentation is consistent with other content in the file):

  <property>
        <name>yarn.nodemanager.pmem-check-enabled</name>
        <value>false</value>
    </property>
    <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
    </property>

distribution of documents

hadoop01:

scp -r /export/servers/hadoop/etc/hadoop hadoop02:/tmp/

hadoop02:

sudo rm -rf /export/servers/hadoop/etc/hadoop
sudo cp -r /tmp/hadoop /export/servers/hadoop/etc/hadoop
rm -r /tmp/hadoop

hadoop01:

scp -r /export/servers/hadoop/etc/hadoop hadoop03:/tmp/

hadoop03:

sudo rm -rf /export/servers/hadoop/etc/hadoop
sudo cp -r /tmp/hadoop /export/servers/hadoop/etc/hadoop
rm -r /tmp/hadoop

After the distribution is completed, each node needs to restart the YARN cluster First enter source /etc/profile to load configuration Enter the /export/servers/spark/sbin catalog

cd /export/servers/spark/sbin

Restart yarn

/export/servers/spark/sbin/stop-yarn.sh
/export/servers/spark/sbin/start-yarn.sh