Spark 集群部署

Spark 的安装并配置启动

下载 Spark 安装包

wget -P /export/software/ https://repo.huaweicloud.com/apache/spark/spark-3.4.3/spark-3.4.3-bin-hadoop3.tgz

解压 Spark 安装包

tar -zxvf /export/software/spark-3.4.3-bin-hadoop3.tgz -C /export/servers
# 重命名:
cd /export/servers
mv /export/servers/spark-3.4.3-bin-hadoop3 /export/servers/spark

修改配置文件

进入spark/conf目录修改 Spark 的配置文件spark-env.sh,将spark-env.sh.template配置模板文件复制一份并命名为spark-env.sh,命令如下:

cd /export/servers/spark/conf
sudo cp /export/servers/spark/conf/spark-env.sh.template /export/servers/spark/conf/spark-env.sh

修改spark-env.sh文件

sudo vim /export/servers/spark/conf/spark-env.sh

在该文件中添加以下内容:

#配置java环境变量
export JAVA_HOME=/export/servers/jdk
#指定Master的IP
export SPARK_MASTER_HOST=hadoop01
#指定Master的端口
export SPARK_MASTER_PORT=7077
export SPARK_MASTER_WEBUI_PORT=8082

复制workers.template文件，并重命名为workers

sudo cp /export/servers/spark/conf/workers.template /export/servers/spark/conf/workers

编辑workers配置文件

sudo vim /export/servers/spark/conf/workers

hadoop01
hadoop02
hadoop03

分发文件

scp -r /export/servers/spark/ hadoop02:/tmp/
scp -r /export/servers/spark/ hadoop03:/tmp/

进入 hadoop02:

sudo cp -r /tmp/spark /export/servers
rm -r /tmp/spark

进入 hadoop03:

sudo cp -r /tmp/spark /export/servers
rm -r /tmp/spark

启动 Spark 集群

分别添加权限

cd /export/servers
sudo chown -R hadooper:hadooper /export/servers/spark/

分步启动 Spark

hadoop01:

cd /export/servers/spark/sbin/
/export/servers/spark/sbin/start-master.sh

hadoop02,hadoop03:

cd /export/servers/spark/sbin/
/export/servers/spark/sbin/start-slave.sh spark://hadoop01:7077

此步的 hadoop02 和 hadoop03 启动时如若出现问题，首先检查主机名是否正确！如若不正确，按照网络上的教程进行修改，此处不做过多赘述

输入$SPARK_HOME/sbin/start-all.sh启动(上述步骤也是启动，为按服务器逐步启动，该启动指令为批量启动),如下如所示即为成功,进一步验证打开本机浏览器输入（hadoop01的IP地址）:8082如果界面正常打开如下所示即代表成功

使 Spark 适配 YARN

配置 Spark 以识别 YARN

进入配置目录/export/servers/spark/conf修改spark-env.sh文件

cd /export/servers/spark/conf
sudo vim /export/servers/spark/conf/spark-env.sh

添加如下内容:

# 设置Hadoop配置文件目录（请根据你的实际路径修改）
export HADOOP_CONF_DIR=/export/servers/hadoop/etc/hadoop

将以下内容注释掉（用#注释）：

export SPARK_MASTER_HOST=hadoop01
export SPARK_MASTER_PORT=7077
export SPARK_MASTER_WEBUI_PORT=8082

调整 YARN 配置

进入配置目录

cd /export/servers/hadoop/etc/hadoop

修改yarn-site.xml文件

sudo vim /export/servers/hadoop/etc/hadoop/yarn-site.xml

添加内容如下(注意缩进要与文件中其他内容保持一致):

  <property>
        <name>yarn.nodemanager.pmem-check-enabled</name>
        <value>false</value>
    </property>
    <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
    </property>

分发文件

hadoop01:

scp -r /export/servers/hadoop/etc/hadoop hadoop02:/tmp/

hadoop02:

sudo rm -rf /export/servers/hadoop/etc/hadoop
sudo cp -r /tmp/hadoop /export/servers/hadoop/etc/hadoop
rm -r /tmp/hadoop

hadoop01:

scp -r /export/servers/hadoop/etc/hadoop hadoop03:/tmp/

hadoop03:

sudo rm -rf /export/servers/hadoop/etc/hadoop
sudo cp -r /tmp/hadoop /export/servers/hadoop/etc/hadoop
rm -r /tmp/hadoop

分发完成后每个节点都需要重启 YARN 集群 先输入source /etc/profile加载配置 进入/export/servers/spark/sbin目录

cd /export/servers/spark/sbin

重新启动 yarn

/export/servers/spark/sbin/stop-yarn.sh
/export/servers/spark/sbin/start-yarn.sh