One-click deployment of Hadoop ecosystem clusters

If you are building a Hadoop family bucket for the first time (ZooKeeper, Hadoop HA, Spark, HBase, Hive, Kafka, Flume, Mahout, Flink), there is a high probability that you have experienced this kind of pain: every component you install requires reading an article, changing a bunch of xml, scp distribution back and forth, and checking from scratch if the startup order is wrong… The whole set will not take a day or two to complete.

this is a setOne-click deployment scripts: Gather all the configurations into one file, upload them to the main node, and run one command. The script will automatically complete the installation, initialization, distribution and startup of all components. There are also menus for daily start-up and stop, self-inspection, and diary reading, so you don’t have to memorize a bunch of commands.

What can this script hold?

With the default three-node hadoop01 / hadoop02 / hadoop03 For example, the role distribution is roughly like this:

• hadoop01 (main node): MySQL, Hive, NameNode (n1, default Active), ResourceManager (rm1, default Active), ZooKeeper, Spark Master, HBase Master, Kafka broker 1, Flume agent, Mahout, Flink client;
• hadoop02 / hadoop03: NameNode (Standby), ResourceManager, JournalNode, DataNode, NodeManager, ZooKeeper, Spark Worker, HBase backup HMaster + RegionServer, Kafka broker 2/3, Flume collector 1/2.

That is to say, the three machines are redundant with each other. If NameNode / ResourceManager hangs, it can automatically switch, and if the Flume main Collector hangs, the data will be switched to the standby Collector.

Working principle (understand in 30 seconds)

infographic sequence-zigzag-steps-underline-text
data
  title Hadoop 集群自动部署流程
  desc 从 Windows 本机上传部署文件，由主节点统一配置并分发至其他节点
  items
    - label Windows 本机
      desc 使用 upload.ps1 / scp 上传 deploy/ 和 installer/

    - label 主节点 hadoop01
      desc 文件统一存放于 /export/deploy/

    - label 统一配置
      desc config/cluster.conf 作为集群唯一配置入口

    - label 执行部署
      desc components/00~90.sh 按顺序执行，并使用 conf/*.tpl 渲染配置

    - label 其他节点
      desc 主节点通过 ssh / scp 分发文件，各节点完成安装和服务启动

theme
  palette
    - #3b82f6
    - #6366f1
    - #8b5cf6
    - #06b6d4
    - #10b981

There are three main things to do inside the script:

1. read configuration: All node, IP, version, port, password, memory parameters are in cluster.conf In a document;
2. rendering configuration: The official configuration files (core-site.xml, hive-site.xml, server.properties, etc.) of each component are made into templates, and scripts are dynamically generated by node;
3. Distribution and execution: The master node distributes the directory to other nodes via SSH (tar streaming), then starts the service one by one and waits for the ports to be ready.

All running logs will write /export/deploy/log/, each component has one file, so you don’t need to rely on guessing when something goes wrong.

Environmental preparation

Recommended to prepare threeNewly installed Ubuntu server(Both apt/yum systems are compatible, but Ubuntu/Debian experience is the best), and satisfy:

• The three machines are interconnected within the intranet, and it is recommended to call the host name directly hadoop01/02/03(The script will also automatically set the host name);
• Each machine has at least an administrator account that can SSH login (Ubuntu prohibits root SSH by default, using the ubuntu Such accounts with sudo permissions);
• Memory should be ≥ 16 GB /node as much as possible (the script defaults to YARN NodeManager 8 GB and Flink TaskManager 4 GB. Remember to lower it for small machines);
• Enough disks: the offline installation package is about 3 GB, and the HDFS data needs to occupy space alone;
• The master node can access the external network apt source; if the installation package is not placed locally, it also needs to be able to access download stations such as Huawei Cloud Image.

Step 1: Modify the cluster configuration

After cloning the code from the repository (or downloading zip directly), open the deploy/config/cluster.conf, this is the only document that needs major changes in the entire process:

# 节点列表与 IP（顺序一一对应）
NODES=(hadoop01 hadoop02 hadoop03)
NODE_IPS=(192.168.52.131 192.168.52.132 192.168.52.133)
MASTER_NODE=hadoop01

# 远程节点的管理员账号（首次 SSH 引导用，三台密码一致）
REMOTE_ROOT_USER="ubuntu"
REMOTE_ROOT_PASSWORD="你的远程管理员密码"

# 脚本会创建的业务用户 hadooper 的密码
HADOOPER_PASSWORD="你的业务用户密码"

# MySQL root 密码、Hive 元数据库账号
MYSQL_ROOT_PASSWORD=root123456
HIVE_DB_NAME=hive
HIVE_DB_USER=hive
HIVE_DB_PASSWORD=hive123456

The following groups can also be adjusted as needed: component version file name (must be and installer/ consistent packages in), each component port, each service startup memory, and component installation directory (default /export/servers）。

ˇThe warehouse comes with a demo password. Please change it to your own if you disclose the warehouse. Don’t submit the real password.

Step 2: Prepare the installation package (optional, but highly recommended)

The script’s installation package strategy is “three-level bottom”:

1. cluster /export/software Already in → directly used;
2. Warehouse local installer/ Directory has → Copy it together when uploading;
3. None → The main node automatically mirrors the image from Huawei Cloud/ Apache official website wget Download.

which means youYou can install it without putting anything down, just downloading big packages such as Hadoop and Flink will be slower. It is recommended to put it in advance cluster.conf Download the zip packages listed in the file name and place them in installer/ Directory can be uploaded once, and deployment speed is much faster.

Step 3: Upload to the master node

Build a directory on the master node first (if you don’t already have it):

sudo mkdir -p /export && sudo chown -R $(whoami) /export

Use the PowerShell scripts that come with the warehouse directly on Windows:

./upload.ps1 -HostName 192.168.52.131 -User root

upload.ps1 will deploy/ and installer/ The entire transmission to the master node /export/ Down,-User Change to an account where you can SSH into the host. If you don’t want to use scripts, scp manually:

scp -r deploy installer ubuntu@192.168.52.131:/export/

Step 4: Start deployment with one command

SSH logs in to the master node and perform:

cd /export/deploy
sudo bash main.sh install

The script automatically completes the 13 stages in the following order:

安装包检查 → 基础环境 → JDK → MySQL → ZooKeeper → Hadoop HA
→ Spark → HBase → Hive → Kafka → Flume → Mahout → Flink

During deployment, you will see real-time logs for each step on the terminal:

• Installation package check: First confirm that all installation packages are available, and automatically download them if they are missing;
• basic environment: Configure hosts and host names, create hadooper user, creation /export/. The three machines were configured with SSH confidentiality immunity, firewall turned off, file handles and process number limits were increased. Note: If the host name changes, the script will prompt you to restart three machines before continuing;
• JDK / MySQL: JDK is distributed to all nodes;MySQL is only installed on the main node, and Hive metadata and accounts are automatically built;
• ZooKeeper: Three units form a cluster, automatically write myid and wait for the election to be completed;
• Hadoop HA: The first installation will automatically format NameNode, initialize QJM shared edits, bootstrap standby, start ZKFC, and disconnect. It takes the longest time here, wait patiently;
• Spark / HBase / Hive: Spark Standalone cluster;HBase master and backup HMaster + RegionServer;Hive automatically initializes metadata schemas and integrates them bi-directionally with Spark (copy dependent jars to HDFS);
• Kafka / Flume: One broker for each node;Flume launches two collectors first and then launches agent, and the Agent collects data on the main node. weblog.log Write to HDFS;
• Mahout / Flink: Mahout is a pure client-side tool;Flink is started in YARN Session mode, ZooKeeper is stored in HA state, and metadata is stored in HDFS.

After all installation, the script will automaticallyStart all services in order of dependence。The entire process lasts from 20 minutes to 1 hour, mainly depending on the installation package download speed and machine performance.

Step 5: Verify deployment results

One-click self-test

sudo bash main.sh status   # 各组件进程/状态总览
sudo bash main.sh test     # 各组件自检

test Instead of simply ping, it’s a real business verification:

• HDFS: Write files, read files, and check HA status;
• Spark: Submit SparkPi;
• HBase: create table → put → get → scan → delete table;
• Hive: Beeline is executed even with HiveServer2 show databases；
• Kafka: Create topics, production, consumption;
• Flume: To weblog.log Write data → Verify HDFS file growth → Intentionally killing the main Collector → Verify that the backup Collector takes over → Automatic recovery;
• Flink: Submit WordCount to the YARN Session and check HDFS output.

browser to verify

Direct access can be made under the default configuration:

Flink’s Web UI can be viewed by clicking on the corresponding Application on the YARN page (8088).

daily operation and maintenance

Don’t remember complex commands, complete them all through the main menu:

cd /export/deploy && sudo bash main.sh

The main menu provides:

1) 一键部署全部       5) 集群状态总览
2) 一键启动全部       6) 安装包检查/补全
3) 一键停止全部       7) 按组件操作（二级菜单）
4) 一键测试全部       8) 查看日志

You can also type the command directly, such as only processing Hadoop:

sudo bash main.sh start                       # 全部启动
sudo bash components/20_hadoop.sh status      # 只看 Hadoop
sudo bash components/50_hive.sh test          # 只测 Hive
sudo bash components/70_flume.sh menu         # 进入 Flume 菜单

Each component script supports install / start / stop / test / status / log / menu These movements, the names are 00_packages.sh to 90_flink.sh, you can match the dependency order by looking at the number.

If there is a problem, read the diary first:

tail -f /export/deploy/log/*.log

Log is divided into files by component:hadoop.log、hive.log、flink.log… You can also choose directly from the menu.

Guide to avoiding pits (all pits that have been dealt with in the script)

1. Ubuntu cannot directly SSH root: For remote nodes, please fill in an ordinary account with sudo permissions, such as ubuntu；
2. Change the host name and restart it: If you prompt the host name change after the basic environment step is completed, restart the three servers first before continuing, otherwise subsequent SSH and hosts will be chaotic;
3. Don’t place the wrong file name in the installation package：cluster.conf in *_TAR Variables must be and installer/ The actual file names in the file are the same, otherwise the script will be downloaded again as missing;
4. Hadoop Don’t swipe and repeat formatting: Script is protected by default (skipping if there is a formatting mark), and only use it if you want to completely rebuild it. FORCE=1 sudo bash components/20_hadoop.sh install；
5. Failure of a single component does not require a full process: After fixing the problem, you can simply rerun the failed component alone, and other components will not be affected;
6. Flume requires at least 3 machines: One agent and two collectors each account for one. If there are less than three, no Flume should be installed;
7. Don’t give too much resources: Default YARN NodeManager is 8GB, Flink TaskManager is 4GB, change first for small memory machines cluster.conf Redeployment of memory parameters in;
8. offline environment: Install the installation package into installer/ Upload together, MySQL and other system packages still require local apt sources.