Spark cluster deployment
Installation and configuration of Spark
Download the Spark installation package
wget -P /export/software/ https://repo.huaweicloud.com/apache/spark/spark-3.4.3/spark-3.4.3-bin-hadoop3.tgzUnpack the Spark installation package
tar -zxvf /export/software/spark-3.4.3-bin-hadoop3.tgz -C /export/servers
# 重命名:
cd /export/servers
mv /export/servers/spark-3.4.3-bin-hadoop3 /export/servers/sparkmodify the configuration file
enterspark/confDirectory Modify Spark’s configuration filespark-env.sh, willspark-env.sh.templateMake a copy of the configuration template file and name itspark-env.sh, the order is as follows:
cd /export/servers/spark/conf
sudo cp /export/servers/spark/conf/spark-env.sh.template /export/servers/spark/conf/spark-env.shmodifyspark-env.shfile
sudo vim /export/servers/spark/conf/spark-env.shAdd the following to the file:
#配置java环境变量
export JAVA_HOME=/export/servers/jdk
#指定Master的IP
export SPARK_MASTER_HOST=hadoop01
#指定Master的端口
export SPARK_MASTER_PORT=7077
export SPARK_MASTER_WEBUI_PORT=8082replicationworkers.templatefile and rename it toworkers
sudo cp /export/servers/spark/conf/workers.template /export/servers/spark/conf/workerseditworkersprofile
sudo vim /export/servers/spark/conf/workershadoop01
hadoop02
hadoop03distribution of documents
scp -r /export/servers/spark/ hadoop02:/tmp/
scp -r /export/servers/spark/ hadoop03:/tmp/Enter hadoop02:
sudo cp -r /tmp/spark /export/servers
rm -r /tmp/sparkEnter hadoop03:
sudo cp -r /tmp/spark /export/servers
rm -r /tmp/sparkLaunch Spark cluster
Add permissions separately
cd /export/servers
sudo chown -R hadooper:hadooper /export/servers/spark/Start Spark step by step
hadoop01:
cd /export/servers/spark/sbin/
/export/servers/spark/sbin/start-master.shhadoop02,hadoop03:
cd /export/servers/spark/sbin/
/export/servers/spark/sbin/start-slave.sh spark://hadoop01:7077If there is a problem when starting hadoop02 and hadoop03 at this step, first check whether the host name is correct! If it is incorrect, modify it according to the tutorial on the Internet, and we will not go into it too much here.
input$SPARK_HOME/sbin/start-all.shStart (the above steps are also start, which is a step by step by server, and this start command is batch start), which means success as shown below. Further verify the input of opening the local browser.（hadoop01的IP地址）:8082If the interface opens normally, as shown below, it means success
Adapting Spark to YARN
Configure Spark to recognize YARN
Enter the configuration directory/export/servers/spark/confmodifyspark-env.shfile
cd /export/servers/spark/conf
sudo vim /export/servers/spark/conf/spark-env.shAdd the following:
# 设置Hadoop配置文件目录（请根据你的实际路径修改）
export HADOOP_CONF_DIR=/export/servers/hadoop/etc/hadoopComment out the following (comment with #):
export SPARK_MASTER_HOST=hadoop01
export SPARK_MASTER_PORT=7077
export SPARK_MASTER_WEBUI_PORT=8082Adjust YARN configuration
Enter the configuration directory
cd /export/servers/hadoop/etc/hadoopmodifyyarn-site.xmlfile
sudo vim /export/servers/hadoop/etc/hadoop/yarn-site.xmlAdd the following content (note that the indentation is consistent with other content in the file):
<property>
<name>yarn.nodemanager.pmem-check-enabled</name>
<value>false</value>
</property>
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>distribution of documents
hadoop01:
scp -r /export/servers/hadoop/etc/hadoop hadoop02:/tmp/hadoop02:
sudo rm -rf /export/servers/hadoop/etc/hadoop
sudo cp -r /tmp/hadoop /export/servers/hadoop/etc/hadoop
rm -r /tmp/hadoophadoop01:
scp -r /export/servers/hadoop/etc/hadoop hadoop03:/tmp/hadoop03:
sudo rm -rf /export/servers/hadoop/etc/hadoop
sudo cp -r /tmp/hadoop /export/servers/hadoop/etc/hadoop
rm -r /tmp/hadoopAfter the distribution is completed, each node needs to restart the YARN cluster
first entersource /etc/profileload configuration
enter/export/servers/spark/sbindirectory
cd /export/servers/spark/sbinRestart yarn
/export/servers/spark/sbin/stop-yarn.sh
/export/servers/spark/sbin/start-yarn.sh
If you enjoyed this, leave a comment~