Hadoop deployment patches

1.The instructions in this document are entered in hadoop01 throughout the entire process (except jps verification)

2.You need to enter the following instructions in hadoop01 to expand the space

sudo lvextend -r -l +100%FREE /dev/ubuntu-vg/ubuntu-lv

Launch hadoop,zookeeper,jobHistory clusters

switch user

su hadooper

Load environment variables

cd ../..

source etc/profile

Launch ZooKeeper

zkServer.sh start

ssh hadoop02 "bash -c 'source /etc/profile && zkServer.sh start'"

ssh hadoop03 "bash -c 'source /etc/profile && zkServer.sh start'"

Launch hadoop

bash /export/servers/hadoop/sbin/start-all.sh

Launch JobHistory

mapred --daemon start historyserver

ssh hadoop02 "bash -c 'source /etc/profile && mapred --daemon start historyserver'"

ssh hadoop03 "bash -c 'source /etc/profile && mapred --daemon start historyserver'"

Remove zookeeper historical relics

Enter ZooKeeper client

zkCli.sh

Remove YARN’s election lock

deleteall /yarn-leader-election

Delete YARN’s status storage

deleteall /rmstore

Check and delete the HA status of HDFS

deleteall /hadoop-ha

Quit ZooKeeper client

quit

replacement file

Both are single lines of instructions

Upload the new yarn-site.xml file to the /export/servers/hadoop/etc/hadoop path through the ssh tool (finalshell\windterm, etc.) to overwrite the old yarn-site.xml file Or modify it through the command line, the steps are as follows:

cd /export/servers/hadoop/etc/hadoop

vim yarn-site.xml

Enter ggdG (strictly according to case) to clear the original content Enter Insert mode, replace the new content with the original file, save and exit Distributed to hadoop02

scp /export/servers/hadoop/etc/hadoop/yarn-site.xml hadoop02:/tmp/ && \

ssh hadoop02 "sudo cp /tmp/yarn-site.xml /export/servers/hadoop/etc/hadoop && sudo rm /tmp/yarn-site.xml"

Distributed to hadoop03

scp /export/servers/hadoop/etc/hadoop/yarn-site.xml hadoop03:/tmp/ && \

ssh hadoop03 "sudo cp /tmp/yarn-site.xml /export/servers/hadoop/etc/hadoop && sudo rm /tmp/yarn-site.xml"

Restart hadoop,zookeeper,jobHistory clusters

Close JobHistory

mapred --daemon stop historyserver

ssh hadoop02 "bash -c 'source /etc/profile && mapred --daemon stop historyserver'"

ssh hadoop03 "bash -c 'source /etc/profile && mapred --daemon stop historyserver'"

Shut down hadoop

bash /export/servers/hadoop/sbin/stop-all.sh

Close ZooKeeper

zkServer.sh stop

ssh hadoop02 "bash -c 'source /etc/profile && zkServer.sh stop'"

ssh hadoop03 "bash -c 'source /etc/profile && zkServer.sh stop'"

Launch ZooKeeper

zkServer.sh start

ssh hadoop02 "bash -c 'source /etc/profile && zkServer.sh start'"

ssh hadoop03 "bash -c 'source /etc/profile && zkServer.sh start'"

Launch hadoop

bash /export/servers/hadoop/sbin/start-all.sh

Reinitialize ZooKeeper

hdfs zkfc -formatZK

Restart the HDFS service

stop-dfs.sh

start-dfs.sh

Launch JobHistory

mapred --daemon start historyserver

ssh hadoop02 "bash -c 'source /etc/profile && mapred --daemon start historyserver'"

ssh hadoop03 "bash -c 'source /etc/profile && mapred --daemon start historyserver'"

After entering jps into all three nodes, there is ResourceManager as shown in the figure. The modification is successful

The ps command needs to load environment variables. The specific operation is also required to switch users in the first part.

Allocate remaining space Enter yarn node -list, as shown in the figure. It may be different, but as long as three nodes are displayed, the startup will be successful.

Enter yarn node -list, as shown in the figure. It may be different, but as long as two standby and one active are displayed, the startup will be successful.