Hadoop version selection
Recommendations for stable versions in production environments
Hadoop 3.4.0
1.This version is a stable distribution of the Apache Hadoop 3.x series. It has long provided bug fixes and compatibility support, and meets the core requirements of most production scenarios. It has significant optimizations in storage efficiency (such as erasure code), resource management and YARN scheduling, making it suitable for enterprise-level production environments. 2.Ecological component compatibility recommended Hadoop 3.4.0 needs to be paired with the following stable versions of ecological components to ensure overall compatibility:
Spark 3.4.3
Features: A general-purpose computing engine based on memory computing, with performance far exceeding MapReduce. Provides libraries such as Spark SQL (Structured Data Processing), Spark Streaming (Stream Processing), and MLlib (Machine Learning) to form a unified stack.
Hive 3.1.3
Features: Convert SQL queries into tasks that run on Hadoop (the default engine is MapReduce or Tez), lowering the threshold for big data queries. Suitable for offline data warehousing and batch processing scenarios.
HBase 2.6.0
Features: A distributed and scalable NoSQL database built on top of HDFS supports random, real-time read and write access to massive amounts of data, making up for HDFS’s shortcomings in low-latency access.
ZooKeeper 3.8.5
Features: Distributed coordination services provide the Hadoop ecosystem with basic capabilities such as reliable distributed locking, leader election, and configuration maintenance. They are the cornerstone of high availability of HDFS and stable operation of HBase and other components.
Kafka 3.4.0
Features: A high-throughput distributed messaging system that decouples producers and consumers and often serves as the hub of a real-time data pipeline.
Flume 1.9.0
Features: Distributed, highly available log collection, aggregation and movement tools that reliably transfer data to HDFS or HBase.
Mahout 14.1
Features: Extensible machine learning algorithm library. Later versions mainly implement distributed machine learning algorithms based on engines such as Spark.
Ambari 2.7.0
Features: Simplify the installation, deployment, configuration and monitoring of Hadoop clusters through a Web interface, which is very suitable for teaching and operation and maintenance management.
JDK 8
It is officially recommended to use JDK 8orJDK 11. In production environments, it is recommended to give priority to JDK 8 that has been supported for a long time (such as 1.8.0_202 and above). If new features are needed, JDK 11 can be used, but component compatibility needs to be tested in advance.
Three-node model requirements
hardware configuration
It is recommended to configure each node with an 8-core CPU, 16GB and above of memory, and 1TB SATA III hard disk (or higher). Nodes are connected through Gigabit and above Ethernet to ensure data transmission efficiency. If the amount of data or calculation pressure is large, you can refer to the following expansion suggestions:
- Master: It is recommended to configure a higher memory (such as 32GB) to run NameNode and ResourceManager.
- Data/compute node (Worker): Configure more disk space (such as 8TB HDD) for data storage.
Systems and dependencies
The operating system requires CentOS 7.x/8.x or Ubuntu 18.04/20.04 LTS version. Install JDK 8 in advance (compatible versions 1.8.0_202 and above are recommended) and complete the following basic configurations:
- Time synchronization: Install NTP services to ensure that cluster nodes have consistent time.
- Network configuration: Set the static IP, modify the host name of each node (such as hadoop001, hadoop002, hadoop003), and configure IP and host name mapping in
/etc/hosts. - SSH secret free login: Configure SSH mutual trust between nodes to ensure that the master node can access all nodes without a password.
- System optimization: Turn off firewall and SELinux (if the production environment needs to be turned on, exceptions need to be set for Hadoop-related ports), and adjust kernel parameters (such as vm.swappiness=10, net.core.somaxconn=65535).
Node role assignment
The following table illustrates the classic 3-node cluster role allocation:
| node host name | address example | main service role | remarks |
|---|---|---|---|
| hadoop001 | 192.168.30.131 | NameNode,ResourceManager, SecondaryNameNode | Master node (management node) |
| hadoop002 | 192.168.30.132 | DataNode,NodeManager | Slave node (data/compute node) |
| hadoop003 | 192.168.30.133 | DataNode, NodeManager | Slave node (data/compute node) |
This architecture concentrates management services (NameNode (Resource Manager) on one node, and two slave nodes undertake the actual data storage and computing tasks. For higher availability, consider deploying the SecondaryNameNode on another node, or expanding to HDFS HA in the future (additional nodes are required to run Standby NameNode and JournalNode).
Key points of cluster deployment and configuration
Critical profile adjustments
The following files need to be configured under the $HADOOP_HOME/etc/hadoop/ directory:
- JVM heap memory adjustment: In configuration files such as
hadoop-env.shandyarn-site.xml, significantly reduce the heap memory limit of each daemon (for example, set the default 1GB or 2GB to 512MB or even 256MB) to make room for application tasks. core-site.xml: Set the default file system (such as fs.defaultFS to hdfs://hadoop001) and the Hadoop temporary directory (hadoop.tmp.dir).hdfs-site.xml: Set the number of copies (dfs.replication) to 1 in hdfs-site.xml because there is only one DataNode in pseudo-distributed mode, saving storage space.yarn-site.xml： Specify the host name of the ResourceManager (yarn.resourcemanager.hostname), and set- - NodeManager auxiliary services (yarn.nodemanager.aux-services are mapreduce_shuffle) to strictly limit the number of physical memory and CPU cores available to NodeManager to prevent a single task from exhausting all resources.workersfile: Add the host names of two slave nodes (hadoop002 and hadoop003) to it.
Cluster initialization and startup
- Format HDFS: Execute
hdfs namenode -formatonly once on the master node. - Start the cluster: Run
start-dfs.shandstart-yarn.shon the main node. - Authentication service: Use the jps command to check whether the process of each node is normal, and check the cluster status through the Web UI (such as NameNode, ResourceManager).
Production environment precautions
- Data balancing: After initial deployment or large amounts of data are written, run hdfs balancer to balance data distribution.
- Logging and monitoring: Configure centralized log collection (such as ELK) and integrate Prometheus and Grafana to monitor key cluster indicators (CPU, memory, HDFS usage, etc.).
- Regular maintenance: Develop a rolling restart strategy, and pay attention to official vulnerability announcements to update patches in a timely manner.
Summary of version selection
For production environments that seek stability, Hadoop 3.4.0+JDK 8+ compatible ecosystem component versions (such asHive 3.1.3,Spark 3.4.3) are a proven and reliable combination. Three-node clusters are a common starting point for learning and small and medium-scale production. Following the above principles of hardware, system configuration and role allocation, a stable and efficient Hadoop operating environment can be built.
If you enjoyed this, leave a comment~