Deployment of Hadoop

Download of hadoop

cd /export/software
wget -P /export/software/ https://repo.huaweicloud.com/apache/hadoop/common/hadoop-3.4.0/hadoop-3.4.0.tar.gz

Decompress the compressed package

tar -zxvf /export/software/hadoop-3.4.0.tar.gz -C /export/servers/
cd /export/servers
mv /export/servers/hadoop-3.4.0 /export/servers/hadoop

Configuration of hadoop

Modify hadoop configuration file

cd /export/servers/hadoop/etc/hadoop

modify`hadoop-env.sh`

sudo vim /export/servers/hadoop/etc/hadoop/hadoop-env.sh

add the following

export JAVA_HOME=/export/servers/jdk

modify`core-site.xml`file

sudo vim /export/servers/hadoop/etc/hadoop/core-site.xml

Add <configuration>the following configurations to the label (strictly indented, at the same level as the original configuration)

 <property>
        <name>fs.defaultFS</name>
        <value>hdfs://mycluster</value>
        <description>HDFS默认文件系统URI</description>
    </property>

    <property>
        <name>hadoop.tmp.dir</name>
        <value>/export/servers/hadoop/tmp</value>
        <description>Hadoop临时目录</description>
    </property>

    <property>
        <name>ha.zookeeper.quorum</name>
        <value>hadoop01:2181,hadoop02:2181,hadoop03:2181</value>
        <description>ZooKeeper仲裁地址，用于自动故障转移</description>
    </property>

    <!-- IO配置优化 -->
    <property>
        <name>io.file.buffer.size</name>
        <value>131072</value>
        <description>读写缓冲区大小</description>
    </property>

modify`hdfs-site.xml`file

sudo vim /export/servers/hadoop/etc/hadoop/hdfs-site.xml

Add <configuration>the following configuration to the label (strictly indented, same level as the original configuration)

  <!-- ========== 基本HDFS配置 ========== -->
    <property>
        <name>dfs.replication</name>
        <value>3</value>
        <description>数据块副本数量，建议与DataNode数量匹配</description>
    </property>

    <!-- ========== 存储目录配置 ========== -->
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:///export/data/hadoop/hdfs/name</value>
        <description>NameNode元数据存储目录</description>
    </property>

    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:///export/data/hadoop/hdfs/data</value>
        <description>DataNode数据块存储目录</description>
    </property>

    <!-- ========== HA 高可用配置 ========== -->
    <property>
        <name>dfs.nameservices</name>
        <value>mycluster</value>
        <description>HDFS命名服务逻辑名称</description>
    </property>

    <property>
        <name>dfs.ha.namenodes.mycluster</name>
        <value>nn1,nn2,nn3</value>
        <description>NameNode逻辑标识列表</description>
    </property>

    <!-- NameNode RPC地址配置 -->
    <property>
        <name>dfs.namenode.rpc-address.mycluster.nn1</name>
        <value>hadoop01:8020</value>
    </property>
    <property>
        <name>dfs.namenode.rpc-address.mycluster.nn2</name>
        <value>hadoop02:8020</value>
    </property>
    <property>
        <name>dfs.namenode.rpc-address.mycluster.nn3</name>
        <value>hadoop03:8020</value>
    </property>

    <!-- NameNode HTTP地址配置 -->
    <property>
        <name>dfs.namenode.http-address.mycluster.nn1</name>
        <value>0.0.0.0:9870</value>
    </property>
    <property>
        <name>dfs.namenode.http-address.mycluster.nn2</name>
        <value>0.0.0.0:9870</value>
    </property>
    <property>
        <name>dfs.namenode.http-address.mycluster.nn3</name>
        <value>0.0.0.0:9870</value>
    </property>

    <!-- JournalNode集群配置 -->
    <property>
        <name>dfs.namenode.shared.edits.dir</name>
        <value>qjournal://hadoop01:8485;hadoop02:8485;hadoop03:8485/mycluster</value>
        <description>JournalNode仲裁地址，用于NameNode元数据同步</description>
    </property>

    <property>
        <name>dfs.journalnode.edits.dir</name>
        <value>/export/servers/hadoop/journalnode</value>
        <description>JournalNode编辑日志存储目录</description>
    </property>

    <!-- 故障转移配置 -->
    <property>
        <name>dfs.client.failover.proxy.provider.mycluster</name>
        <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
        <description>故障转移代理提供者类</description>
    </property>

    <property>
        <name>dfs.ha.automatic-failover.enabled</name>
        <value>true</value>
        <description>启用自动故障转移</description>
    </property>

    <!-- 故障防护配置 -->
    <property>
        <name>dfs.ha.fencing.methods</name>
        <value>sshfence</value>
        <description>故障防护方法，防止脑裂</description>
    </property>

    <property>
        <name>dfs.ha.fencing.ssh.private-key-files</name>
        <value>/home/hadooper/.ssh/id_rsa</value>
        <description>SSH防护使用的私钥文件</description>
    </property>

    <!-- ========== 性能优化配置 ========== -->
    <property>
        <name>dfs.blocksize</name>
        <value>134217728</value>
        <description>HDFS块大小，默认128MB</description>
    </property>

    <property>
        <name>dfs.namenode.handler.count</name>
        <value>100</value>
        <description>NameNode处理线程数</description>
    </property>

    <!-- ========== Web UI 安全配置 ========== -->
    <property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
        <description>启用WebHDFS REST API</description>
    </property>

modify`mapred-site.xml`file

sudo cp /export/servers/hadoop/etc/hadoop/mapred-site.xml /export/servers/hadoop/etc/hadoop/mapred-site.xml.template
sudo vim /export/servers/hadoop/etc/hadoop/mapred-site.xml

Add <configuration>the following configurations to the label (strictly indented, at the same level as the original configuration)

    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>

modify`yarn-site.xml`file

sudo vim /export/servers/hadoop/etc/hadoop/yarn-site.xml

Add <configuration>the following configurations to the label (strictly indented, at the same level as the original configuration)

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>
	<!-- ========== ResourceManager 核心配置 ========== -->
	<!-- ResourceManager Web UI地址 -->
	<property>
		<name>yarn.resourcemanager.webapp.address.rm1</name>
		<value>hadoop01:8088</value>
	</property>
	<property>
	        <name>yarn.resourcemanager.webapp.address.rm2</name>
		<value>hadoop02:8088</value>
	</property>
	<property>
		<name>yarn.resourcemanager.webapp.address.rm3</name>
		<value>hadoop03:8088</value>
	</property>
	<!-- ResourceManager对客户端服务的地址 -->
	<property>
		<name>yarn.resourcemanager.address.rm1</name>
		<value>hadoop01:8032</value>
	</property>
        <property>
		<name>yarn.resourcemanager.address.rm2</name>
		<value>hadoop02:8032</value>
	</property>
	<property>
		<name>yarn.resourcemanager.address.rm3</name>
		<value>hadoop03:8032</value>
	</property>
	<!-- ResourceManager对管理端服务的地址 -->
	<property>
		<name>yarn.resourcemanager.admin.address.rm1</name>
		<value>hadoop01:8033</value>
	</property>
	<property>
		<name>yarn.resourcemanager.admin.address.rm2</name>
		<value>hadoop02:8033</value>
	</property>
	<property>
		<name>yarn.resourcemanager.admin.address.rm3</name>
		<value>hadoop03:8033</value>
	</property>						
	<!-- ResourceManager调度器地址 -->
	<property>
		<name>yarn.resourcemanager.scheduler.address.rm1</name>
		<value>hadoop01:8030</value>
	</property>
	<property>
		<name>yarn.resourcemanager.scheduler.address.rm2</name>
		<value>hadoop02:8030</value>
	</property>
	<property>
		<name>yarn.resourcemanager.scheduler.address.rm3</name>
		<value>hadoop03:8030</value>
	</property>

	<!-- ResourceManager资源追踪器地址 -->
	<property>
		<name>yarn.resourcemanager.resource-tracker.address.rm1</name>
		<value>hadoop01:8031</value>
	</property>
	<property>
		<name>yarn.resourcemanager.resource-tracker.address.rm2</name>
		<value>hadoop02:8031</value>
	</property>
	<property>
		<name>yarn.resourcemanager.resource-tracker.address.rm3</name>
		<value>hadoop03:8031</value>
	</property>
	<!-- 指定 ZooKeeper 集群地址 -->
	<property>
		<name>yarn.resourcemanager.zk-address</name>
		<value>hadoop01:2181,hadoop02:2181,hadoop03:2181</value>
	</property>
	<!-- ========== ResourceManager HA 核心配置 ========== -->
	<property>
		<name>yarn.resourcemanager.ha.enabled</name>
		<value>true</value>
	</property>
	<!-- 指定YARN集群的逻辑ID -->
	<property>
		<name>yarn.resourcemanager.cluster-id</name>
		<value>cluster-yarn</value>
		<description>YARN集群的逻辑标识符</description>
	</property>
	<!-- 指定ResourceManager的逻辑ID列表，用逗号分隔 -->
	<property>
		<name>yarn.resourcemanager.ha.rm-ids</name>
		<value>rm1,rm2,rm3</value>
		<description>定义集群中ResourceManager的逻辑名称</description>
	</property>
	<property>
		<name>yarn.resourcemanager.hostname.rm1</name>
		<value>hadoop01</value>
	</property>
	<property>
		<name>yarn.resourcemanager.hostname.rm2</name>
		<value>hadoop02</value>
	</property>
	<property>
		<name>yarn.resourcemanager.hostname.rm3</name>
		<value>hadoop03</value>
	</property>
	<!-- ========== NodeManager 核心配置 ========== -->
	<!-- NodeManager的辅助服务，必须设置为mapreduce_shuffle -->
	<property>
		<name>yarn.nodemanager.aux-services</name>
		<value>mapreduce_shuffle</value>
		<description>NodeManager的辅助服务列表，mapreduce_shuffle是MapReduce必需的</description>
	</property>
	<!-- 指定shuffle处理的类 -->
	<property>
		<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
		<value>org.apache.hadoop.mapred.ShuffleHandler</value>
		<description>shuffle服务的实现类</description>
	</property>
	<!-- ========== 资源管理配置 ========== -->
	<!-- NodeManager可用的物理内存资源 -->
	<property>
		<name>yarn.nodemanager.resource.memory-mb</name>
		<value>8192</value>
		<description>NodeManager可分配给容器的物理内存总量（MB）</description>
	</property>
	<!-- NodeManager可用的CPU核心数 -->
	<property>
		<name>yarn.nodemanager.resource.cpu-vcores</name>
		<value>4</value>
		<description>NodeManager可分配给容器的虚拟CPU核心数</description>
	</property>
	<!-- 单个容器可申请的最小内存 -->
	<property>
		<name>yarn.scheduler.minimum-allocation-mb</name>
		<value>1024</value>
		<description>调度器分配给每个容器请求的最小内存（MB）</description>
	</property>
	<!-- 单个容器可申请的最大内存 -->
	<property>
		<name>yarn.scheduler.maximum-allocation-mb</name>
		<value>8192</value>
		<description>调度器分配给每个容器请求的最大内存（MB）</description>
	</property>
	<!-- ========== 日志聚合配置 ========== -->
	<!-- 启用日志聚合功能 -->
	<property>
		<name>yarn.log-aggregation-enable</name>
		<value>true</value>
		<description>是否启用日志聚合，启用后容器日志会汇总到HDFS</description>
	</property>
	<!-- 日志在HDFS中的保留时间 -->
	<property>
		<name>yarn.log-aggregation.retain-seconds</name>
		<value>2592000</value>
		<description>聚合日志在HDFS中的保留时间（秒），默认30天</description>
	</property>
	<!-- ========== NodeManager Web UI配置 ========== -->
	<!-- NodeManager Web UI地址 -->
	<property>
		<name>yarn.nodemanager.webapp.address</name>
		<value>0.0.0.0:8042</value>
		<description>NodeManager的Web界面地址</description>
	</property>
	<!-- 远程日志目录 -->
	<property>
		<name>yarn.nodemanager.remote-app-log-dir</name>
		<value>/tmp/logs</value>
		<description>应用程序日志在HDFS中的存储目录</description>
	</property>

	<property>
		<name>yarn.nodemanager.pmem-check-enabled</name>
		<value>false</value>
	</property>
	<property>
		<name>yarn.nodemanager.vmem-check-enabled</name>
		<value>false</value>
	</property>

<!-- Site specific YARN configuration properties -->

</configuration>

modify`workers`file

sudo vim /export/servers/hadoop/etc/hadoop/workers

Change to the following

hadoop01
hadoop02
hadoop03

Distribution of hadoop

Configure hadoop environment variables

Configure three server hadoop system environment variables separately

sudo bash -c 'cat >> /etc/profile << "EOF"
export HADOOP_HOME=/export/servers/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
EOF'

ssh hadoop02 "sudo tee -a /etc/profile << 'EOF'
export HADOOP_HOME=/export/servers/hadoop
export PATH=\$PATH:/export/servers/hadoop/bin:/export/servers/hadoop/sbin
EOF"

ssh hadoop03 "sudo tee -a /etc/profile << 'EOF'
export HADOOP_HOME=/export/servers/hadoop
export PATH=\$PATH:/export/servers/hadoop/bin:/export/servers/hadoop/sbin
EOF"

Distribute hadoop profiles

scp -r /export/servers/hadoop hadoop02:/tmp/
ssh hadoop02 "sudo cp -r /tmp/hadoop/ /export/servers/ && sudo rm -rf /tmp/hadoop"
scp -r /export/servers/hadoop hadoop03:/tmp/
ssh hadoop03 "sudo cp -r /tmp/hadoop/ /export/servers/ && sudo rm -rf /tmp/hadoop"

Verify that hadoop was successfully installed

source /etc/profile
hadoop version
ssh hadoop02 "bash -c 'source /etc/profile && hadoop version'"
ssh hadoop03 "bash -c 'source /etc/profile && hadoop version'"

Operation of hadoop

Changes to hadoop running environment variables

sudo bash -c 'cat >> /etc/profile << EOF
# Hadoop环境变量配置
# 配置HDFS相关进程的运行用户
export HDFS_NAMENODE_USER=hadooper
export HDFS_DATANODE_USER=hadooper
export HDFS_SECONDARYNAMENODE_USER=hadooper
# 配置YARN相关进程的运行用户
export YARN_RESOURCEMANAGER_USER=hadooper
export YARN_NODEMANAGER_USER=hadooper
EOF'

ssh hadoop02 "sudo bash -c 'cat >> /etc/profile << \"EOF\"
# Hadoop环境变量配置
# 配置HDFS相关进程的运行用户
export HDFS_NAMENODE_USER=hadooper
export HDFS_DATANODE_USER=hadooper
export HDFS_SECONDARYNAMENODE_USER=hadooper
# 配置YARN相关进程的运行用户
export YARN_RESOURCEMANAGER_USER=hadooper
export YARN_NODEMANAGER_USER=hadooper
EOF'"

ssh hadoop03 "sudo bash -c 'cat >> /etc/profile << \"EOF\"
# Hadoop环境变量配置
# 配置HDFS相关进程的运行用户
export HDFS_NAMENODE_USER=hadooper
export HDFS_DATANODE_USER=hadooper
export HDFS_SECONDARYNAMENODE_USER=hadooper
# 配置YARN相关进程的运行用户
export YARN_RESOURCEMANAGER_USER=hadooper
export YARN_NODEMANAGER_USER=hadooper
EOF'"

Reload environment variables

source /etc/profile
ssh hadoop02 "bash -c 'source /etc/profile'"
ssh hadoop03 "bash -c 'source /etc/profile'"

The final environment variable configuration is as follows

Launched hadoop for the first time

Create necessary folders

sudo mkdir -p /export/servers/hadoop/logs
sudo chown -R hadooper /export/servers/hadoop/
sudo chmod -R 755 /export/servers/hadoop/
sudo mkdir -p /export/data/hadoop/
sudo chown -R hadooper /export/data/hadoop/
sudo chmod -R 755 /export/data/hadoop/
ssh hadoop02 "sudo mkdir -p /export/servers/hadoop/logs && sudo chown -R hadooper /export/servers/hadoop/ && sudo chmod -R 755 /export/servers/hadoop/ && sudo mkdir -p /export/data/hadoop/ && sudo chown -R hadooper /export/data/hadoop/ && sudo chmod -R 755 /export/data/hadoop/"
ssh hadoop03 "sudo mkdir -p /export/servers/hadoop/logs && sudo chown -R hadooper /export/servers/hadoop/ && sudo chmod -R 755 /export/servers/hadoop/ && sudo mkdir -p /export/data/hadoop/ && sudo chown -R hadooper /export/data/hadoop/ && sudo chmod -R 755 /export/data/hadoop/"

Launch ZooKeeper

$ZOOKEEPER_HOME/bin/zkServer.sh start
ssh hadoop02 "bash -c 'source /etc/profile && $ZOOKEEPER_HOME/bin/zkServer.sh start'"
ssh hadoop03 "bash -c 'source /etc/profile && $ZOOKEEPER_HOME/bin/zkServer.sh start'"

Launch journalnode

hdfs --daemon start journalnode
ssh hadoop02 "bash -c 'source /etc/profile && hdfs --daemon start journalnode'"
ssh hadoop03 "bash -c 'source /etc/profile && hdfs --daemon start journalnode'"

This step needs to be run on server hadoop01 if and only if the file system is initialized before starting hadoop for the first time

hdfs namenode -format

This code can only be operated when the hdfs cluster is started for the first time!

Start namenode

hdfs namenode -initializeSharedEdits
hdfs --daemon start namenode
ssh hadoop02 "bash -c 'source /etc/profile && hdfs namenode -bootstrapStandby'"
ssh hadoop03 "bash -c 'source /etc/profile && hdfs namenode -bootstrapStandby'"

Execute the run script

cd /export/servers/hadoop/sbin
bash /export/servers/hadoop/sbin/start-all.sh

Launch ZKFC

hdfs zkfc -formatZK
hdfs --daemon start zkfc
ssh hadoop02 "bash -c 'source /etc/profile && hdfs --daemon start zkfc'"
ssh hadoop03 "bash -c 'source /etc/profile && hdfs --daemon start zkfc'"

Launch JobHistory

mapred --daemon start historyserver
ssh hadoop02 "bash -c 'source /etc/profile && mapred --daemon start historyserver'"
ssh hadoop03 "bash -c 'source /etc/profile && mapred --daemon start historyserver'"

Verify that each part started successfully

jps
ssh hadoop02 "bash -c 'source /etc/profile && jps'"
ssh hadoop03 "bash -c 'source /etc/profile && jps'"

Subsequent activation and shutdown of hadoop

Launch ZooKeeper

$ZOOKEEPER_HOME/bin/zkServer.sh start
ssh hadoop02 "bash -c 'source /etc/profile && $ZOOKEEPER_HOME/bin/zkServer.sh start'"
ssh hadoop03 "bash -c 'source /etc/profile && $ZOOKEEPER_HOME/bin/zkServer.sh start'"

Launch hadoop

bash /export/servers/hadoop/sbin/start-all.sh

Launch JobHistory

mapred --daemon start historyserver
ssh hadoop02 "bash -c 'source /etc/profile && mapred --daemon start historyserver'"
ssh hadoop03 "bash -c 'source /etc/profile && mapred --daemon start historyserver'"

Close JobHistory

mapred --daemon stop historyserver
ssh hadoop02 "bash -c 'source /etc/profile && mapred --daemon stop historyserver'"
ssh hadoop03 "bash -c 'source /etc/profile && mapred --daemon stop historyserver'"

Shut down hadoop

bash /export/servers/hadoop/sbin/stop-all.sh

Close ZooKeeper

$ZOOKEEPER_HOME/bin/zkServer.sh stop
ssh hadoop02 "bash -c 'source /etc/profile && $ZOOKEEPER_HOME/bin/zkServer.sh stop'"
ssh hadoop03 "bash -c 'source /etc/profile && $ZOOKEEPER_HOME/bin/zkServer.sh stop'"

test

The browser accesses the following URLs (IP needs to be replaced)

DataNode Information

http://10.1.100.20:9864
http://10.1.100.21:9864
http://10.1.100.22:9864

NodeManager information

http://10.1.100.20:8042
http://10.1.100.21:8042
http://10.1.100.22:8042

Namenode information

http://10.1.100.20:9870/

Nodes of the cluster

http://10.1.100.20:8088/

JobHistory

http://10.1.100.20:19888/jobhistory/
http://10.1.100.21:19888/jobhistory/
http://10.1.100.22:19888/jobhistory/

Successful access and content means successful launch

Hadoop deployments

Deployment of Hadoop

Download of hadoop

Configuration of hadoop

modify`hadoop-env.sh`

modify`core-site.xml`file

modify`hdfs-site.xml`file

modify`mapred-site.xml`file

modify`yarn-site.xml`file

modify`workers`file

Distribution of hadoop

Configure hadoop environment variables

Distribute hadoop profiles

Verify that hadoop was successfully installed

Operation of hadoop

Changes to hadoop running environment variables

Reload environment variables

Launched hadoop for the first time

Subsequent activation and shutdown of hadoop

test

Random Posts

Related Posts

Hadoop deployments

Deployment of Hadoop

Download of hadoop

Configuration of hadoop

modifyhadoop-env.sh

modifycore-site.xmlfile

modifyhdfs-site.xmlfile

modifymapred-site.xmlfile

modifyyarn-site.xmlfile

modifyworkersfile

Distribution of hadoop

Configure hadoop environment variables

Distribute hadoop profiles

Verify that hadoop was successfully installed

Operation of hadoop

Changes to hadoop running environment variables

Reload environment variables

Launched hadoop for the first time

Subsequent activation and shutdown of hadoop

test

Random Posts

Related Posts

modify`hadoop-env.sh`

modify`core-site.xml`file

modify`hdfs-site.xml`file

modify`mapred-site.xml`file

modify`yarn-site.xml`file

modify`workers`file