心得技巧
html5HTML/XhtmlCSSXML/XSLTDreamweaver教程Frontpage教程心得技巧

zookeeper

发布时间：2022-07-02 发布网站：脚本宝典
脚本宝典收集整理的这篇文章主要介绍了zookeeper，脚本宝典觉得挺不错的，现在分享给大家，也给大家做个参考。

Top

NSD ArchITECTURE DAY05

  1. 案例1：组建zookeePEr集群
  2. 案例2：测试集群的远程管理和高可用
  3. 案例3：在node节点上搭建3台kafka
  4. 案例4：准备实验环境
  5. 案例5：配置namenode与resourcemanager高可用
  6. 案例6：启动服务，验证高可用

1 案例1：组建zookeeper集群

1.1 问题

本案例要求：

  • 组建 zookeeper 集群
  • 1 个 leader
  • 2 个 follower
  • 1 个 observer

1.2 步骤

实现此案例需要按照如下步骤进行。

步骤一：安装Zookeeper

1）编辑/etc/hosts ,所有集群主机可以相互 ping 通（在hadoop1上面配置，同步到node-0001，node-0002，node-0003）

 
  1. [root@hadoop1 hadoop]# vim /etc/hosts
  2. 192.168.1.50 hadoop1
  3. 192.168.1.51 node-0001
  4. 192.168.1.52 node-0002
  5. 192.168.1.53 node-0003
  6. 192.168.1.56 newnode
  8. [root@nn01 hadoop]# for i in {52..54}
  9. do    
  10. scp /etc/hosts 192.168.1.$i:/etc/    
  11. done        //同步配置
  12. hosts 100% 253 639.2KB/s 00:00
  13. hosts 100% 253 497.7KB/s 00:00
  14. hosts 100% 253 662.2KB/s 00:00    

2）安装 java-1.8.0-openjdk-devel,由于之前的hadoop上面已经安装过，这里不再安装，若是新机器要安装

3）zookeeper 解压拷贝到 /usr/local/zookeeper

  1. [root@hadoop1 ~]# tar -xf zookeeper-3.4.13.tar.gz
  2. [root@hadoop1 ~]# mv zookeeper-3.4.13 /usr/local/zookeeper

4）配置文件改名，并在最后添加配置

  1. [root@hadoop1 ~]# cd /usr/local/zookeeper/conf/
  2. [root@hadoop1 conf]# ls
  3. configuration.xsl LOG4j.PRoperties zoo_sample.CFg
  4. [root@hadoop1 conf]# mv zoo_sample.cfg zoo.cfg
  5. [root@hadoop1 conf]# chown root.root zoo.cfg
  6. [root@hadoop1 conf]# vim zoo.cfg
  7. server.1=node-0001:2888:3888
  8. server.2=node-0002:2888:3888
  9. server.3=node-0003:2888:3888
  10. server.4=hadoop1:2888:3888:observer

5）拷贝 /usr/local/zookeeper 到其他集群主机

  1. [root@hadoop1 conf]# for i in {52..54}; do rsync -aSH --delete /usr/local/zookeeper/ 192.168.1.$i:/usr/local/zookeeper -e 'ssh' & done
  2. [4] 4956
  3. [5] 4957
  4. [6] 4958

6）创建 mkdir /tmp/zookeeper，每一台都要

  1. [root@hadoop1 conf]# mkdir /tmp/zookeeper
  2. [root@hadoop1 conf]# ssh node-0001 mkdir /tmp/zookeeper
  3. [root@hadoop1 conf]# ssh node-0002 mkdir /tmp/zookeeper
  4. [root@hadoop1 conf]# ssh node-0003 mkdir /tmp/zookeeper

7）创建 myid 文件，id 必须与配置文件里主机名对应的 server.(id) 一致

  1. [root@hadoop1 conf]# echo 4 >/tmp/zookeeper/myid
  2. [root@hadoop1 conf]# ssh node-0001 'echo 1 >/tmp/zookeeper/myid'
  3. [root@hadoop1 conf]# ssh node-0002 'echo 2 >/tmp/zookeeper/myid'
  4. [root@hadoop1 conf]# ssh node-0003 'echo 3 >/tmp/zookeeper/myid'

8）启动服务，单启动一台无法查看状态，需要启动全部集群以后才能查看状态，每一台上面都要手工启动（以hadoop1为例子）

  1. [root@hadoop1 conf]# /usr/local/zookeeper/bin/zkServer.sh start
  2. ZooKeeper JMX enabled by default
  3. Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
  4. Starting zookeeper ... STARTED

注意：刚启动zookeeper查看状态的时候报错，启动的数量要保证半数以上，这时再去看就成功了

9）查看状态

  1. [root@hadoop1 conf]# /usr/local/zookeeper/bin/zkServer.sh status
  2. ZooKeeper JMX enabled by default
  3. Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
  4. Mode: observe
  5. [root@hadoop1 conf]# /usr/local/zookeeper/bin/zkServer.sh stop
  6. //关闭之后查看状态其他服务器的角色
  7. ZooKeeper JMX enabled by default
  8. Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
  9. Stopping zookeeper ... STOPPED

2 案例2：测试集群的远程管理和高可用

2.1 问题

本案例要求：

  • 测试集群的远程管理和高可用

2.2 步骤

实现此案例需要按照如下步骤进行。

步骤一： 测试集群的远程管理和高可用

  1. [root@hadoop1 conf]# socat - TCP:node1:2181
  2. stat
  3. ... ...
  4. Outstanding: 0
  5. Zxid: 0x0
  6. Mode: follower
  7. Node count: 4
  8. [root@hadoop1 conf]# vim api.sh
  9. #!/bin/bash
  10. function getstatus(){
  11. exec 9<>/dev/tcp/$1/2181 2>/dev/null
  12. echo stat >&9
  13. MODE=$(cat <&9 |grep -Po "(?<=Mode:).*")
  14. exec 9<&-
  15. echo ${MODE:-NULL}
  16. }
  17. for i in node{1..3} hadoop1;do
  18. echo -ne "${i}t"
  19. getstatus ${i}
  20. done
  21. [root@hadoop1 conf]# chmod 755 api.sh
  22. [root@hadoop1 conf]# ./api.sh
  23. node-0001        follower
  24. node-0002        leader
  25. node-0003        follower
  26. hadoop1        observer

3 案例3：在node节点上搭建3台kafka

3.1 问题

本案例要求：

  • 在node节点上搭建3台kafka
  • node-0001
  • node-0002
  • node-0003
  • 发布订阅消息测试

3.2 步骤

实现此案例需要按照如下步骤进行。

步骤一：搭建Kafka集群

1）解压 kafka 压缩包

Kafka在node-0001，node-0002，node-0003上面操作即可

  1. [root@node-0001 hadoop]# tar -xf kafka_2.12-2.1.0.tgz

2）把 kafka 拷贝到 /usr/local/kafka 下面

  1. [root@node-0001 ~]# mv kafka_2.12-2.1.0 /usr/local/kafka

3）修改配置文件 /usr/local/kafka/config/server.properties

  1. [root@node-0001 ~]# cd /usr/local/kafka/config
  2. [root@node-0001 config]# vim server.properties
  3. broker.id=22
  4. zookeeper.connect=node-0001:2181,node-0002:2181,node-0003:2181

4）拷贝 kafka 到其他主机，并修改 broker.id ,不能重复

  1. [root@node-0001 config]# for i in 53 54; do rsync -aSH --delete /usr/local/kafka 192.168.1.$i:/usr/local/; done
  2. [1] 27072
  3. [2] 27073
  5. [root@node-0002 ~]# vim /usr/local/kafka/config/server.properties        
  6. //node-0002主机修改
  7. broker.id=23
  8. [root@node-0003 ~]# vim /usr/local/kafka/config/server.properties        
  9. //node-0003主机修改
  10. broker.id=24

5）启动 kafka 集群（node-0001，node-0002，node-0003启动）

  1. [root@node-0001 local]# /usr/local/kafka/bin/kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties
  2. [root@node-0001 local]# jps        //出现kafka
  3. 26483 DataNode
  4. 27859 Jps
  5. 27833 Kafka
  6. 26895 QuorumPeerMain

6）验证配置，创建一个 topic

  1. [root@node-0001 local]# /usr/local/kafka/bin/kafka-topics.sh --create --partitions 1 --replication-factor 1 --zookeeper localhost:2181 --topic mymsg
  2.     
  3. Created topic "mymsg".

7) 模拟生产者，发布消息

  1. [root@node-0002 ~]# /usr/local/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic mymsg
  2.         //写一个数据
  3. ccc
  4. ddd

9）模拟消费者，接收消息

  1. [root@node-0003 ~]# /usr/local/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic mymsg
  2.         //这边会直接同步
  3. ccc
  4. ddd

注意：kafka比较吃内存，做完这个kafka的实验可以把它停了

4 案例4：准备实验环境

4.1 问题

本案例要求：

  • 所有节点
  • 192.168.1.50 hadoop1
  • 192.168.1.56 hadoop2
  • 192.168.1.51 node-0001
  • 192.168.1.52 node-0002
  • 192.168.1.53 node-0003
  • 新机器安装 java-1.8.0-openjdk-devel
  • 新机器配置 /etc/hosts
  • 新机器配置 ssh 免密钥登录
  • 修改配置文件

4.2 方案

配置Hadoop的高可用，解决NameNode单点故障问题，使用之前搭建好的hadoop集群，新添加一台hadoop2，ip为192.168.1.56，具体要求如图-1所示：

zookeeper

图-1

4.3 步骤

实现此案例需要按照如下步骤进行。

步骤一：hadoop的高可用

1）停止所有服务（由于 kafka的实验做完之后就已经停止，这里不在重复）

  1. [root@hadoop1 ~]# cd /usr/local/hadoop/
  2. [root@hadoop1 hadoop]# ./sbin/stop-all.sh //停止所有服务

2）启动zookeeper（需要一台一台的启动）这里以hadoop1为例子

  1. [root@hadoop1 hadoop]# /usr/local/zookeeper/bin/zkServer.sh start
  2. [root@hadoop1 hadoop]# sh /usr/local/zookeeper/conf/api.sh //利用之前写好的脚本查看
  3. node-0001        follower
  4. node-0002        leader
  5. node-0003        follower
  6. hadoop1        observer

3）新加一台机器hadoop2，这里之前有一台newnode，可以用这个作为hadoop2

  1. [root@newnode ~]# echo hadoop2 > /etc/hostname
  2. [root@newnode ~]# hostname hadoop2

4）修改vim /etc/hosts

  1. [root@hadoop1 hadoop]# vim /etc/hosts
  2. 192.168.1.50 hadoop1
  3. 192.168.1.56 hadoop2
  4. 192.168.1.51 node-0001
  5. 192.168.1.52 node-0002
  6. 192.168.1.53 node-0003

5）同步到hadoop2，node-0001，node-0002，node-0003

  1. [root@hadoop1 hadoop]# for i in {51..53} 56; do rsync -aSH --delete /etc/hosts 192.168.1.$i:/etc/hosts -e 'ssh' & done
  2. [1] 14355
  3. [2] 14356
  4. [3] 14357
  5. [4] 14358

6）配置SSH信任关系

注意：hadoop1和hadoop2互相连接不需要密码，hadoop2连接自己和node-0001，node-0002，node-0003同样不需要密码

  1. [root@hadoop2 ~]# vim /etc/ssh/ssh_config
  2. Host *
  3. GSSAPIAuthentication yes
  4. StrictHostKeyChecking no
  5. [root@hadoop1 hadoop]# cd /root/.ssh/
  6. [root@hadoop1 .ssh]# scp id_rsa id_rsa.pub hadoop2:/root/.ssh/    
  7. //把hadoop1的公钥私钥考给hadoop2

7）所有的主机删除/VAR/hadoop/*

  1. [root@hadoop1 .ssh]# rm -rf /var/hadoop/*

5 案例5：配置namenode与resourcemanager高可用

5.1 问题

本案例要求：

  • 配置 namenode 与 resourcemanager 高可用
  • hadoop-env.sh
  • core-site.XMl
  • hDFs-site.xML
  • mapred-site.xml
  • yarn-site.xml
  • slaves

5.2 步骤

实现此案例需要按照如下步骤进行。

步骤一：hadoop的高可用

1）配置 core-site

  1. [root@hadoop1 .ssh]# vim /usr/local/hadoop/etc/hadoop/core-site.xml
  2. <configuration>
  3. <property>
  4. <name>fs.defaultFS</name>
  5. <value>hdfs://nsdcluster</value>    
  6. //nsdcluster是随便起的名。相当于一个组，访问的时候访问这个组
  7. </property>
  8. <property>
  9. <name>hadoop.tmp.dir</name>
  10. <value>/var/hadoop</value>
  11. </property>
  12. <property>
  13. <name>ha.zookeeper.quorum</name>
  14. <value>node-0001:2181,node-0002:2181,node-0003:2181</value>    //zookeepe的地址
  15. </property>
  16. <property>
  17. <name>hadoop.Proxyuser.nfs.groups</name>
  18. <value>*</value>
  19. </property>
  20. <property>
  21. <name>hadoop.proxyuser.nfs.hosts</name>
  22. <value>*</value>
  23. </property>
  24. </configuration>

2）配置 hdfs-site

  1. [root@hadoop1 ~]# vim /usr/local/hadoop/etc/hadoop/hdfs-site.xml
  2. <configuration>
  3. <property>
  4. <name>dfs.replication</name>
  5. <value>2</value>
  6. </property>
  7. <property>
  8. <name>dfs.nameservices</name>
  9. <value>nsdcluster</value>
  10. </property>
  11. <property>
  12. <name>dfs.ha.namenodes.nsdcluster</name>                
  13. //nn1,nn2名称固定，是内置的变量，nsdcluster里面有nn1，nn2
  14. <value>nn1,nn2</value>
  15. </property>
  16. <property>
  17. <name>dfs.namenode.rpc-address.nsdcluster.nn1</name>        
  18. //声明nn1 8020为通讯端口，是hadoop1的rpc通讯端口
  19. <value>hadoop1:8020</value>
  20. </property>
  21. <property>
  22. <name>dfs.namenode.rpc-address.nsdcluster.nn2</name>        
  23. //声明nn2是谁，hadoop2的rpc通讯端口
  24. <value>hadoop2:8020</value>
  25. </property>
  26. <property>
  27. <name>dfs.namenode.http-address.nsdcluster.nn1</name>    
  28. //hadoop1的http通讯端口
  29. <value>hadoop1:50070</value>
  30. </property>
  31. <property>
  32. <name>dfs.namenode.http-address.nsdcluster.nn2</name>     
  33. //hadoop1和hadoop2的http通讯端口
  34. <value>hadoop2:50070</value>
  35. </property>
  36. <property>
  37. <name>dfs.namenode.shared.edits.dir</name>        
  38. //指定namenode元数据存储在journalnode中的路径
  39. <value>qjournal://node-0001:8485;node-0002:8485;node-0003:8485/nsdcluster</value>
  40. </property>
  41. <property>
  42. <name>dfs.journalnode.edits.dir</name>            
  43. //指定journalnode日志文件存储的路径
  44. <value>/var/hadoop/journal</value>
  45. </property>
  46. <property>
  47. <name>dfs.client.failover.proxy.provider.nsdcluster</name>    
  48. //指定HDFS客户端连接active namenode的java类
  49. <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
  50. </property>
  51. <property>
  52. <name>dfs.ha.fencing.methods</name>                    //配置隔离机制为ssh
  53. <value>sshfence</value>
  54. </property>
  55. <property>
  56. <name>dfs.ha.fencing.ssh.private-key-files</name>    //指定密钥的位置
  57. <value>/root/.ssh/id_rsa</value>
  58. </property>
  59. <property>
  60. <name>dfs.ha.automatic-failover.enabled</name>        //开启自动故障转移
  61. <value>true</value>                
  62. </property>
  63. </configuration>

3）配置yarn-site

  1. [root@hadoop1 ~]# vim /usr/local/hadoop/etc/hadoop/yarn-site.xml
  2. <configuration>
  4. <!-- Site specific YARN configuration properties -->
  5. <property>
  6. <name>yarn.nodemanager.aux-services</name>
  7. <value>;mapreduce_shuffle</value>
  8. </property>
  9. <property>
  10. <name>yarn.resourcemanager.ha.enabled</name>
  11. <value>true</value>
  12. </property>
  13. <property>
  14. <name>yarn.resourcemanager.ha.rm-ids</name>        //rm1,rm2代表hadoop1和hadoop2
  15. <value>rm1,rm2</value>
  16. </property>
  17. <property>
  18. <name>yarn.resourcemanager.recovery.enabled</name>
  19. <value>true</value>
  20. </property>
  21. <property>
  22. <name>yarn.resourcemanager.Store.class</name>
  23. <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStatestore</value>
  24. </property>
  25. <property>
  26. <name>yarn.resourcemanager.zk-address</name>
  27. <value>node-0001:2181,node-0002:2181,node-0003:2181</value>
  28. </property>
  29. <property>
  30. <name>yarn.resourcemanager.cluster-id</name>
  31. <value>yarn-ha</value>
  32. </property>
  33. <property>
  34. <name>yarn.resourcemanager.hostname.rm1</name>
  35. <value>hadoop1</value>
  36. </property>
  37. <property>
  38. <name>yarn.resourcemanager.hostname.rm2</name>
  39. <value>hadoop2</value>
  40. </property>
  41. </configuration>

6 案例6：启动服务，验证高可用

6.1 问题

本案例要求：

  • 启动服务，验证高可用
  • 分析数据时停止一个活跃节点
  • 验证高可用状态及数据

6.2 步骤

实现此案例需要按照如下步骤进行。

步骤一：hadoop高可用的验证

1）同步到hadoop2，node-0001，node-0002，node-0003

  1. [root@hadoop1 ~]# for i in {51..53} 56; do rsync -aSH --delete /usr/local/hadoop/ 192.168.1.$i:/usr/local/hadoop -e 'ssh' & done
  2. [1] 25411
  3. [2] 25412
  4. [3] 25413
  5. [4] 25414

2）删除所有机器上面的/user/local/hadoop/logs，方便排错

  1. [root@hadoop1 ~]# for i in {50..53} 56; do ssh 192.168.1.$i rm -rf /usr/local/hadoop/logs ; done

3）同步配置

  1. [root@hadoop1 ~]# for i in {51..53} 56; do rsync -aSH --delete /usr/local/hadoop 192.168.1.$i:/usr/local/hadoop -e 'ssh' & done
  3. [1] 28235
  4. [2] 28236
  5. [3] 28237
  6. [4] 28238

4）初始化ZK集群

  1. [root@hadoop1 ~]# /usr/local/hadoop/bin/hdfs zkfc -formatZK
  2. ...
  3. 18/09/11 15:43:35 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/nsdcluster in ZK //出现Successfully即为成功
  4. ...

5）在node-0001，node-0002，node-0003上面启动journalnode服务（以node-0001为例子）

  1. [root@node-0001 ~]# /usr/local/hadoop/sbin/hadoop-daemon.sh start journalnode
  2. starting journalnode, logging to /usr/local/hadoop/logs/hadoop-root-journalnode-node-0001.out
  3. [root@node-0001 ~]# jps
  4. 29262 JournalNode
  5. 26895 QuorumPeerMain
  6. 29311 Jps

6）格式化，先在node-0001，node-0002，node-0003上面启动journalnode才能格式化

  1. [root@hadoop1 ~]# /usr/local/hadoop//bin/hdfs namenode -format
  2. //出现Successfully即为成功
  3. [root@hadoop1 hadoop]# ls /var/hadoop/
  4. dfs

7）hadoop2数据同步到本地 /var/hadoop/dfs

  1. [root@hadoop2 ~]# cd /var/hadoop/
  2. [root@hadoop2 hadoop]# ls
  3. [root@hadoop2 hadoop]# rsync -aSH hadoop1:/var/hadoop/ /var/hadoop/
  4. [root@hadoop2 hadoop]# ls
  5. dfs

8）初始化 JNS

  1. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/hdfs namenode -initializeSharedEdits
  2. 18/09/11 16:26:15 INFO client.QuorumJournalManager: Successfully started new epoch 1        //出现Successfully，成功开启一个节点

9）停止 journalnode 服务（node-0001，node-0002，node-0003）

  1. [root@node-0001 hadoop]# /usr/local/hadoop/sbin/hadoop-daemon.sh stop journalnode
  2. stopping journalnode
  3. [root@node-0001 hadoop]# jps
  4. 29346 Jps
  5. 26895 QuorumPeerMain

步骤二：启动集群

1）hadoop1上面操作

  1. [root@hadoop1 hadoop]# /usr/local/hadoop/sbin/start-all.sh //启动所有集群
  2. This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
  3. Starting namenodes on [hadoop1 hadoop2]
  4. hadoop1: starting namenode, logging to /usr/local/hadoop/logs/hadoop-root-namenode-hadoop1.out
  5. hadoop2: starting namenode, logging to /usr/local/hadoop/logs/hadoop-root-namenode-hadoop2.out
  6. node-0002: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-node-0002.out
  7. node-0003: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-node-0003.out
  8. node-0001: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-node-0001.out
  9. Starting journal nodes [node-0001 node-0002 node-0003]
  10. node-0001: starting journalnode, logging to /usr/local/hadoop/logs/hadoop-root-journalnode-node-0001.out
  11. node-0003: starting journalnode, logging to /usr/local/hadoop/logs/hadoop-root-journalnode-node-0003.out
  12. node-0002: starting journalnode, logging to /usr/local/hadoop/logs/hadoop-root-journalnode-node-0002.out
  13. Starting ZK Failover Controllers on NN hosts [hadoop1 hadoop2]
  14. hadoop1: starting zkfc, logging to /usr/local/hadoop/logs/hadoop-root-zkfc-hadoop1.out
  15. hadoop2: starting zkfc, logging to /usr/local/hadoop/logs/hadoop-root-zkfc-hadoop2.out
  16. starting yarn daemons
  17. starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-root-resourcemanager-hadoop1.out
  18. node-0002: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-node-0002.out
  19. node-0001: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-node-0001.out
  20. node-0003: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-node-0003.out

2）hadoop2上面操作

  1. [root@hadoop2 hadoop]# /usr/local/hadoop/sbin/yarn-daemon.sh start resourcemanager
  2. starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-root-resourcemanager-hadoop2.out

3）查看集群状态

  1. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/hdfs haadmin -getServiceState nn1
  2. active
  3. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/hdfs haadmin -getServiceState nn2
  4. standby
  5. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/yarn rmadmin -getServiceState rm1
  6. active
  7. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/yarn rmadmin -getServiceState rm2
  8. standby

4）查看节点是否加入

  1. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/hdfs dfsadmin -report
  2. ...
  3. Live datanodes (3): //会有三个节点
  4. ...
  5. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/yarn node -list
  6. total Nodes:3
  7. Node-Id     Node-State    Node-Http-Address    Number-of-Running-Containers
  8. node-0002:43307     RUNNING     node-0002:8042     0
  9. node-0001:34606     RUNNING     node-0001:8042     0
  10. node-0003:36749     RUNNING     node-0003:8042     0

步骤三：访问集群

1）查看并创建

  1. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/hadoop fs -ls /
  2. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/hadoop fs -mkdir /aa //创建aa
  3. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/hadoop fs -ls /        //再次查看
  5. Found 1 items
  6. drwxr-xr-x - root supergroup 0 2018-09-11 16:54 /aa
  8. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/hadoop fs -put *.txt /aa
  9. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/hadoop fs -ls hdfs://nsdcluster/aa
  10. //也可以这样查看
  11. Found 3 items
  12. -rw-r--r-- 2 root supergroup 86424 2018-09-11 17:00 hdfs://nsdcluster/aa/LICENSE.txt
  13. -rw-r--r-- 2 root supergroup 14978 2018-09-11 17:00 hdfs://nsdcluster/aa/NOTICE.txt
  14. -rw-r--r-- 2 root supergroup 1366 2018-09-11 17:00 hdfs://nsdcluster/aa/README.txt

2）验证高可用，关闭 actiTop

NSD ARCHITECTURE DAY05

  1. 案例1：组建zookeeper集群
  2. 案例2：测试集群的远程管理和高可用
  3. 案例3：在node节点上搭建3台kafka
  4. 案例4：准备实验环境
  5. 案例5：配置namenode与resourcemanager高可用
  6. 案例6：启动服务，验证高可用

1 案例1：组建zookeeper集群

1.1 问题

本案例要求：

  • 组建 zookeeper 集群
  • 1 个 leader
  • 2 个 follower
  • 1 个 observer

1.2 步骤

实现此案例需要按照如下步骤进行。

步骤一：安装Zookeeper

1）编辑/etc/hosts ,所有集群主机可以相互 ping 通（在hadoop1上面配置，同步到node-0001，node-0002，node-0003）

 
  1. [root@hadoop1 hadoop]# vim /etc/hosts
  2. 192.168.1.50 hadoop1
  3. 192.168.1.51 node-0001
  4. 192.168.1.52 node-0002
  5. 192.168.1.53 node-0003
  6. 192.168.1.56 newnode
  8. [root@nn01 hadoop]# for i in {52..54}
  9. do    
  10. scp /etc/hosts 192.168.1.$i:/etc/    
  11. done        //同步配置
  12. hosts 100% 253 639.2KB/s 00:00
  13. hosts 100% 253 497.7KB/s 00:00
  14. hosts 100% 253 662.2KB/s 00:00    

2）安装 java-1.8.0-openjdk-devel,由于之前的hadoop上面已经安装过，这里不再安装，若是新机器要安装

3）zookeeper 解压拷贝到 /usr/local/zookeeper

  1. [root@hadoop1 ~]# tar -xf zookeeper-3.4.13.tar.gz
  2. [root@hadoop1 ~]# mv zookeeper-3.4.13 /usr/local/zookeeper

4）配置文件改名，并在最后添加配置

  1. [root@hadoop1 ~]# cd /usr/local/zookeeper/conf/
  2. [root@hadoop1 conf]# ls
  3. configuration.xsl log4j.properties zoo_sample.cfg
  4. [root@hadoop1 conf]# mv zoo_sample.cfg zoo.cfg
  5. [root@hadoop1 conf]# chown root.root zoo.cfg
  6. [root@hadoop1 conf]# vim zoo.cfg
  7. server.1=node-0001:2888:3888
  8. server.2=node-0002:2888:3888
  9. server.3=node-0003:2888:3888
  10. server.4=hadoop1:2888:3888:observer

5）拷贝 /usr/local/zookeeper 到其他集群主机

  1. [root@hadoop1 conf]# for i in {52..54}; do rsync -aSH --delete /usr/local/zookeeper/ 192.168.1.$i:/usr/local/zookeeper -e 'ssh' & done
  2. [4] 4956
  3. [5] 4957
  4. [6] 4958

6）创建 mkdir /tmp/zookeeper，每一台都要

  1. [root@hadoop1 conf]# mkdir /tmp/zookeeper
  2. [root@hadoop1 conf]# ssh node-0001 mkdir /tmp/zookeeper
  3. [root@hadoop1 conf]# ssh node-0002 mkdir /tmp/zookeeper
  4. [root@hadoop1 conf]# ssh node-0003 mkdir /tmp/zookeeper

7）创建 myid 文件，id 必须与配置文件里主机名对应的 server.(id) 一致

  1. [root@hadoop1 conf]# echo 4 >/tmp/zookeeper/myid
  2. [root@hadoop1 conf]# ssh node-0001 'echo 1 >/tmp/zookeeper/myid'
  3. [root@hadoop1 conf]# ssh node-0002 'echo 2 >/tmp/zookeeper/myid'
  4. [root@hadoop1 conf]# ssh node-0003 'echo 3 >/tmp/zookeeper/myid'

8）启动服务，单启动一台无法查看状态，需要启动全部集群以后才能查看状态，每一台上面都要手工启动（以hadoop1为例子）

  1. [root@hadoop1 conf]# /usr/local/zookeeper/bin/zkServer.sh start
  2. ZooKeeper JMX enabled by default
  3. Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
  4. Starting zookeeper ... STARTED

注意：刚启动zookeeper查看状态的时候报错，启动的数量要保证半数以上，这时再去看就成功了

9）查看状态

  1. [root@hadoop1 conf]# /usr/local/zookeeper/bin/zkServer.sh status
  2. ZooKeeper JMX enabled by default
  3. Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
  4. Mode: observe
  5. [root@hadoop1 conf]# /usr/local/zookeeper/bin/zkServer.sh stop
  6. //关闭之后查看状态其他服务器的角色
  7. ZooKeeper JMX enabled by default
  8. Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
  9. Stopping zookeeper ... STOPPED

2 案例2：测试集群的远程管理和高可用

2.1 问题

本案例要求：

  • 测试集群的远程管理和高可用

2.2 步骤

实现此案例需要按照如下步骤进行。

步骤一： 测试集群的远程管理和高可用

  1. [root@hadoop1 conf]# socat - TCP:node1:2181
  2. stat
  3. ... ...
  4. Outstanding: 0
  5. Zxid: 0x0
  6. Mode: follower
  7. Node count: 4
  8. [root@hadoop1 conf]# vim api.sh
  9. #!/bin/bash
  10. function getstatus(){
  11. exec 9<>/dev/tcp/$1/2181 2>/dev/null
  12. echo stat >&9
  13. MODE=$(cat <&9 |grep -Po "(?<=Mode:).*")
  14. exec 9<&-
  15. echo ${MODE:-NULL}
  16. }
  17. for i in node{1..3} hadoop1;do
  18. echo -ne "${i}t"
  19. getstatus ${i}
  20. done
  21. [root@hadoop1 conf]# chmod 755 api.sh
  22. [root@hadoop1 conf]# ./api.sh
  23. node-0001        follower
  24. node-0002        leader
  25. node-0003        follower
  26. hadoop1        observer

3 案例3：在node节点上搭建3台kafka

3.1 问题

本案例要求：

  • 在node节点上搭建3台kafka
  • node-0001
  • node-0002
  • node-0003
  • 发布订阅消息测试

3.2 步骤

实现此案例需要按照如下步骤进行。

步骤一：搭建Kafka集群

1）解压 kafka 压缩包

Kafka在node-0001，node-0002，node-0003上面操作即可

  1. [root@node-0001 hadoop]# tar -xf kafka_2.12-2.1.0.tgz

2）把 kafka 拷贝到 /usr/local/kafka 下面

  1. [root@node-0001 ~]# mv kafka_2.12-2.1.0 /usr/local/kafka

3）修改配置文件 /usr/local/kafka/config/server.properties

  1. [root@node-0001 ~]# cd /usr/local/kafka/config
  2. [root@node-0001 config]# vim server.properties
  3. broker.id=22
  4. zookeeper.connect=node-0001:2181,node-0002:2181,node-0003:2181

4）拷贝 kafka 到其他主机，并修改 broker.id ,不能重复

  1. [root@node-0001 config]# for i in 53 54; do rsync -aSH --delete /usr/local/kafka 192.168.1.$i:/usr/local/; done
  2. [1] 27072
  3. [2] 27073
  5. [root@node-0002 ~]# vim /usr/local/kafka/config/server.properties        
  6. //node-0002主机修改
  7. broker.id=23
  8. [root@node-0003 ~]# vim /usr/local/kafka/config/server.properties        
  9. //node-0003主机修改
  10. broker.id=24

5）启动 kafka 集群（node-0001，node-0002，node-0003启动）

  1. [root@node-0001 local]# /usr/local/kafka/bin/kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties
  2. [root@node-0001 local]# jps        //出现kafka
  3. 26483 DataNode
  4. 27859 Jps
  5. 27833 Kafka
  6. 26895 QuorumPeerMain

6）验证配置，创建一个 topic

  1. [root@node-0001 local]# /usr/local/kafka/bin/kafka-topics.sh --create --partitions 1 --replication-factor 1 --zookeeper localhost:2181 --topic mymsg
  2.     
  3. Created topic "mymsg".

7) 模拟生产者，发布消息

  1. [root@node-0002 ~]# /usr/local/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic mymsg
  2.         //写一个数据
  3. ccc
  4. ddd

9）模拟消费者，接收消息

  1. [root@node-0003 ~]# /usr/local/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic mymsg
  2.         //这边会直接同步
  3. ccc
  4. ddd

注意：kafka比较吃内存，做完这个kafka的实验可以把它停了

4 案例4：准备实验环境

4.1 问题

本案例要求：

  • 所有节点
  • 192.168.1.50 hadoop1
  • 192.168.1.56 hadoop2
  • 192.168.1.51 node-0001
  • 192.168.1.52 node-0002
  • 192.168.1.53 node-0003
  • 新机器安装 java-1.8.0-openjdk-devel
  • 新机器配置 /etc/hosts
  • 新机器配置 ssh 免密钥登录
  • 修改配置文件

4.2 方案

配置Hadoop的高可用，解决NameNode单点故障问题，使用之前搭建好的hadoop集群，新添加一台hadoop2，ip为192.168.1.56，具体要求如图-1所示：

zookeeper

图-1

4.3 步骤

实现此案例需要按照如下步骤进行。

步骤一：hadoop的高可用

1）停止所有服务（由于 kafka的实验做完之后就已经停止，这里不在重复）

  1. [root@hadoop1 ~]# cd /usr/local/hadoop/
  2. [root@hadoop1 hadoop]# ./sbin/stop-all.sh //停止所有服务

2）启动zookeeper（需要一台一台的启动）这里以hadoop1为例子

  1. [root@hadoop1 hadoop]# /usr/local/zookeeper/bin/zkServer.sh start
  2. [root@hadoop1 hadoop]# sh /usr/local/zookeeper/conf/api.sh //利用之前写好的脚本查看
  3. node-0001        follower
  4. node-0002        leader
  5. node-0003        follower
  6. hadoop1        observer

3）新加一台机器hadoop2，这里之前有一台newnode，可以用这个作为hadoop2

  1. [root@newnode ~]# echo hadoop2 > /etc/hostname
  2. [root@newnode ~]# hostname hadoop2

4）修改vim /etc/hosts

  1. [root@hadoop1 hadoop]# vim /etc/hosts
  2. 192.168.1.50 hadoop1
  3. 192.168.1.56 hadoop2
  4. 192.168.1.51 node-0001
  5. 192.168.1.52 node-0002
  6. 192.168.1.53 node-0003

5）同步到hadoop2，node-0001，node-0002，node-0003

  1. [root@hadoop1 hadoop]# for i in {51..53} 56; do rsync -aSH --delete /etc/hosts 192.168.1.$i:/etc/hosts -e 'ssh' & done
  2. [1] 14355
  3. [2] 14356
  4. [3] 14357
  5. [4] 14358

6）配置SSH信任关系

注意：hadoop1和hadoop2互相连接不需要密码，hadoop2连接自己和node-0001，node-0002，node-0003同样不需要密码

  1. [root@hadoop2 ~]# vim /etc/ssh/ssh_config
  2. Host *
  3. GSSAPIAuthentication yes
  4. StrictHostKeyChecking no
  5. [root@hadoop1 hadoop]# cd /root/.ssh/
  6. [root@hadoop1 .ssh]# scp id_rsa id_rsa.pub hadoop2:/root/.ssh/    
  7. //把hadoop1的公钥私钥考给hadoop2

7）所有的主机删除/var/hadoop/*

  1. [root@hadoop1 .ssh]# rm -rf /var/hadoop/*

5 案例5：配置namenode与resourcemanager高可用

5.1 问题

本案例要求：

  • 配置 namenode 与 resourcemanager 高可用
  • hadoop-env.sh
  • core-site.xml
  • hdfs-site.xml
  • mapred-site.xml
  • yarn-site.xml
  • slaves

5.2 步骤

实现此案例需要按照如下步骤进行。

步骤一：hadoop的高可用

1）配置 core-site

  1. [root@hadoop1 .ssh]# vim /usr/local/hadoop/etc/hadoop/core-site.xml
  2. <configuration>
  3. <property>
  4. <name>fs.defaultFS</name>
  5. <value>hdfs://nsdcluster</value>    
  6. //nsdcluster是随便起的名。相当于一个组，访问的时候访问这个组
  7. </property>
  8. <property>
  9. <name>hadoop.tmp.dir</name>
  10. <value>/var/hadoop</value>
  11. </property>
  12. <property>
  13. <name>ha.zookeeper.quorum</name>
  14. <value>node-0001:2181,node-0002:2181,node-0003:2181</value>    //zookeepe的地址
  15. </property>
  16. <property>
  17. <name>hadoop.proxyuser.nfs.groups</name>
  18. <value>*</value>
  19. </property>
  20. <property>
  21. <name>hadoop.proxyuser.nfs.hosts</name>
  22. <value>*</value>
  23. </property>
  24. </configuration>

2）配置 hdfs-site

  1. [root@hadoop1 ~]# vim /usr/local/hadoop/etc/hadoop/hdfs-site.xml
  2. <configuration>
  3. <property>
  4. <name>dfs.replication</name>
  5. <value>2</value>
  6. </property>
  7. <property>
  8. <name>dfs.nameservices</name>
  9. <value>nsdcluster</value>
  10. </property>
  11. <property>
  12. <name>dfs.ha.namenodes.nsdcluster</name>                
  13. //nn1,nn2名称固定，是内置的变量，nsdcluster里面有nn1，nn2
  14. <value>nn1,nn2</value>
  15. </property>
  16. <property>
  17. <name>dfs.namenode.rpc-address.nsdcluster.nn1</name>        
  18. //声明nn1 8020为通讯端口，是hadoop1的rpc通讯端口
  19. <value>hadoop1:8020</value>
  20. </property>
  21. <property>
  22. <name>dfs.namenode.rpc-address.nsdcluster.nn2</name>        
  23. //声明nn2是谁，hadoop2的rpc通讯端口
  24. <value>hadoop2:8020</value>
  25. </property>
  26. <property>
  27. <name>dfs.namenode.http-address.nsdcluster.nn1</name>    
  28. //hadoop1的http通讯端口
  29. <value>hadoop1:50070</value>
  30. </property>
  31. <property>
  32. <name>dfs.namenode.http-address.nsdcluster.nn2</name>     
  33. //hadoop1和hadoop2的http通讯端口
  34. <value>hadoop2:50070</value>
  35. </property>
  36. <property>
  37. <name>dfs.namenode.shared.edits.dir</name>        
  38. //指定namenode元数据存储在journalnode中的路径
  39. <value>qjournal://node-0001:8485;node-0002:8485;node-0003:8485/nsdcluster</value>
  40. </property>
  41. <property>
  42. <name>dfs.journalnode.edits.dir</name>            
  43. //指定journalnode日志文件存储的路径
  44. <value>/var/hadoop/journal</value>
  45. </property>
  46. <property>
  47. <name>dfs.client.failover.proxy.provider.nsdcluster</name>    
  48. //指定HDFS客户端连接active namenode的java类
  49. <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
  50. </property>
  51. <property>
  52. <name>dfs.ha.fencing.methods</name>                    //配置隔离机制为ssh
  53. <value>sshfence</value>
  54. </property>
  55. <property>
  56. <name>dfs.ha.fencing.ssh.private-key-files</name>    //指定密钥的位置
  57. <value>/root/.ssh/id_rsa</value>
  58. </property>
  59. <property>
  60. <name>dfs.ha.automatic-failover.enabled</name>        //开启自动故障转移
  61. <value>true</value>                
  62. </property>
  63. </configuration>

3）配置yarn-site

  1. [root@hadoop1 ~]# vim /usr/local/hadoop/etc/hadoop/yarn-site.xml
  2. <configuration>
  4. <!-- Site specific YARN configuration properties -->
  5. <property>
  6. <name>yarn.nodemanager.aux-services</name>
  7. <value>mapreduce_shuffle</value>
  8. </property>
  9. <property>
  10. <name>yarn.resourcemanager.ha.enabled</name>
  11. <value>true</value>
  12. </property>
  13. <property>
  14. <name>yarn.resourcemanager.ha.rm-ids</name>        //rm1,rm2代表hadoop1和hadoop2
  15. <value>rm1,rm2</value>
  16. </property>
  17. <property>
  18. <name>yarn.resourcemanager.recovery.enabled</name>
  19. <value>true</value>
  20. </property>
  21. <property>
  22. <name>yarn.resourcemanager.store.class</name>
  23. <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
  24. </property>
  25. <property>
  26. <name>yarn.resourcemanager.zk-address</name>
  27. <value>node-0001:2181,node-0002:2181,node-0003:2181</value>
  28. </property>
  29. <property>
  30. <name>yarn.resourcemanager.cluster-id</name>
  31. <value>yarn-ha</value>
  32. </property>
  33. <property>
  34. <name>yarn.resourcemanager.hostname.rm1</name>
  35. <value>hadoop1</value>
  36. </property>
  37. <property>
  38. <name>yarn.resourcemanager.hostname.rm2</name>
  39. <value>hadoop2</value>
  40. </property>
  41. </configuration>

6 案例6：启动服务，验证高可用

6.1 问题

本案例要求：

  • 启动服务，验证高可用
  • 分析数据时停止一个活跃节点
  • 验证高可用状态及数据

6.2 步骤

实现此案例需要按照如下步骤进行。

步骤一：hadoop高可用的验证

1）同步到hadoop2，node-0001，node-0002，node-0003

  1. [root@hadoop1 ~]# for i in {51..53} 56; do rsync -aSH --delete /usr/local/hadoop/ 192.168.1.$i:/usr/local/hadoop -e 'ssh' & done
  2. [1] 25411
  3. [2] 25412
  4. [3] 25413
  5. [4] 25414

2）删除所有机器上面的/user/local/hadoop/logs，方便排错

  1. [root@hadoop1 ~]# for i in {50..53} 56; do ssh 192.168.1.$i rm -rf /usr/local/hadoop/logs ; done

3）同步配置

  1. [root@hadoop1 ~]# for i in {51..53} 56; do rsync -aSH --delete /usr/local/hadoop 192.168.1.$i:/usr/local/hadoop -e 'ssh' & done
  3. [1] 28235
  4. [2] 28236
  5. [3] 28237
  6. [4] 28238

4）初始化ZK集群

  1. [root@hadoop1 ~]# /usr/local/hadoop/bin/hdfs zkfc -formatZK
  2. ...
  3. 18/09/11 15:43:35 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/nsdcluster in ZK //出现Successfully即为成功
  4. ...

5）在node-0001，node-0002，node-0003上面启动journalnode服务（以node-0001为例子）

  1. [root@node-0001 ~]# /usr/local/hadoop/sbin/hadoop-daemon.sh start journalnode
  2. starting journalnode, logging to /usr/local/hadoop/logs/hadoop-root-journalnode-node-0001.out
  3. [root@node-0001 ~]# jps
  4. 29262 JournalNode
  5. 26895 QuorumPeerMain
  6. 29311 Jps

6）格式化，先在node-0001，node-0002，node-0003上面启动journalnode才能格式化

  1. [root@hadoop1 ~]# /usr/local/hadoop//bin/hdfs namenode -format
  2. //出现Successfully即为成功
  3. [root@hadoop1 hadoop]# ls /var/hadoop/
  4. dfs

7）hadoop2数据同步到本地 /var/hadoop/dfs

  1. [root@hadoop2 ~]# cd /var/hadoop/
  2. [root@hadoop2 hadoop]# ls
  3. [root@hadoop2 hadoop]# rsync -aSH hadoop1:/var/hadoop/ /var/hadoop/
  4. [root@hadoop2 hadoop]# ls
  5. dfs

8）初始化 JNS

  1. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/hdfs namenode -initializeSharedEdits
  2. 18/09/11 16:26:15 INFO client.QuorumJournalManager: Successfully started new epoch 1        //出现Successfully，成功开启一个节点

9）停止 journalnode 服务（node-0001，node-0002，node-0003）

  1. [root@node-0001 hadoop]# /usr/local/hadoop/sbin/hadoop-daemon.sh stop journalnode
  2. stopping journalnode
  3. [root@node-0001 hadoop]# jps
  4. 29346 Jps
  5. 26895 QuorumPeerMain

步骤二：启动集群

1）hadoop1上面操作

  1. [root@hadoop1 hadoop]# /usr/local/hadoop/sbin/start-all.sh //启动所有集群
  2. This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
  3. Starting namenodes on [hadoop1 hadoop2]
  4. hadoop1: starting namenode, logging to /usr/local/hadoop/logs/hadoop-root-namenode-hadoop1.out
  5. hadoop2: starting namenode, logging to /usr/local/hadoop/logs/hadoop-root-namenode-hadoop2.out
  6. node-0002: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-node-0002.out
  7. node-0003: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-node-0003.out
  8. node-0001: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-node-0001.out
  9. Starting journal nodes [node-0001 node-0002 node-0003]
  10. node-0001: starting journalnode, logging to /usr/local/hadoop/logs/hadoop-root-journalnode-node-0001.out
  11. node-0003: starting journalnode, logging to /usr/local/hadoop/logs/hadoop-root-journalnode-node-0003.out
  12. node-0002: starting journalnode, logging to /usr/local/hadoop/logs/hadoop-root-journalnode-node-0002.out
  13. Starting ZK Failover Controllers on NN hosts [hadoop1 hadoop2]
  14. hadoop1: starting zkfc, logging to /usr/local/hadoop/logs/hadoop-root-zkfc-hadoop1.out
  15. hadoop2: starting zkfc, logging to /usr/local/hadoop/logs/hadoop-root-zkfc-hadoop2.out
  16. starting yarn daemons
  17. starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-root-resourcemanager-hadoop1.out
  18. node-0002: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-node-0002.out
  19. node-0001: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-node-0001.out
  20. node-0003: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-node-0003.out

2）hadoop2上面操作

  1. [root@hadoop2 hadoop]# /usr/local/hadoop/sbin/yarn-daemon.sh start resourcemanager
  2. starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-root-resourcemanager-hadoop2.out

3）查看集群状态

  1. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/hdfs haadmin -getServiceState nn1
  2. active
  3. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/hdfs haadmin -getServiceState nn2
  4. standby
  5. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/yarn rmadmin -getServiceState rm1
  6. active
  7. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/yarn rmadmin -getServiceState rm2
  8. standby

4）查看节点是否加入

  1. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/hdfs dfsadmin -report
  2. ...
  3. Live datanodes (3): //会有三个节点
  4. ...
  5. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/yarn node -list
  6. Total Nodes:3
  7. Node-Id     Node-State    Node-Http-Address    Number-of-Running-Containers
  8. node-0002:43307     RUNNING     node-0002:8042     0
  9. node-0001:34606     RUNNING     node-0001:8042     0
  10. node-0003:36749     RUNNING     node-0003:8042     0

步骤三：访问集群

1）查看并创建

  1. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/hadoop fs -ls /
  2. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/hadoop fs -mkdir /aa //创建aa
  3. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/hadoop fs -ls /        //再次查看
  5. Found 1 items
  6. drwxr-xr-x - root supergroup 0 2018-09-11 16:54 /aa
  8. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/hadoop fs -put *.txt /aa
  9. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/hadoop fs -ls hdfs://nsdcluster/aa
  10. //也可以这样查看
  11. Found 3 items
  12. -rw-r--r-- 2 root supergroup 86424 2018-09-11 17:00 hdfs://nsdcluster/aa/LICENSE.txt
  13. -rw-r--r-- 2 root supergroup 14978 2018-09-11 17:00 hdfs://nsdcluster/aa/NOTICE.txt
  14. -rw-r--r-- 2 root supergroup 1366 2018-09-11 17:00 hdfs://nsdcluster/aa/README.txt

2）验证高可用，关闭 active namenode

  1. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/hdfs haadmin -getServiceState nn1
  2. active
  3. [root@hadoop1 hadoop]# /usr/local/hadoop/sbin/hadoop-daemon.sh stop namenode
  4. stopping namenode
  5. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/hdfs haadmin -getServiceState nn1
  6. //再次查看会报错
  7. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/hdfs haadmin -getServiceState nn2
  8. //hadoop2由之前的standby变为active
  9. active
  11. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/yarn rmadmin -getServiceState rm1
  12. active
  13. [root@hadoop1 hadoop]# /usr/local/hadoop/sbin/yarn-daemon.sh stop resourcemanager
  14. //停止resourcemanager
  15. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/yarn rmadmin -getServiceState rm2
  16. active

3） 恢复节点

  1. [root@hadoop1 hadoop]# /usr/local/hadoop/sbin/hadoop-daemon.sh start namenode
  2. //启动namenode
  3. [root@hadoop1 hadoop]# /usr/local/hadoop/sbin/yarn-daemon.sh start resourcemanager
  4. //启动resourcemanager
  5. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/hdfs haadmin -getServiceState nn1
  6. //查看
  7. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/yarn rmadmin -getServiceState rm1
  8. //查看

ve namenode

  1. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/hdfs haadmin -getServiceState nn1
  2. active
  3. [root@hadoop1 hadoop]# /usr/local/hadoop/sbin/hadoop-daemon.sh stop namenode
  4. stopping namenode
  5. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/hdfs haadmin -getServiceState nn1
  6. //再次查看会报错
  7. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/hdfs haadmin -getServiceState nn2
  8. //hadoop2由之前的standby变为active
  9. active
  11. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/yarn rmadmin -getServiceState rm1
  12. active
  13. [root@hadoop1 hadoop]# /usr/local/hadoop/sbin/yarn-daemon.sh stop resourcemanager
  14. //停止resourcemanager
  15. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/yarn rmadmin -getServiceState rm2
  16. active

3） 恢复节点

  1. [root@hadoop1 hadoop]# /usr/local/hadoop/sbin/hadoop-daemon.sh start namenode
  2. //启动namenode
  3. [root@hadoop1 hadoop]# /usr/local/hadoop/sbin/yarn-daemon.sh start resourcemanager
  4. //启动resourcemanager
  5. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/hdfs haadmin -getServiceState nn1
  6. //查看
  7. [root@hadoop1 hadoop]# /usr/local/hadoop/bin/yarn rmadmin -getServiceState rm1
  8. //查看

脚本宝典总结

以上是脚本宝典为你收集整理的zookeeper全部内容，希望文章能够帮你解决zookeeper所遇到的问题。

如果觉得脚本宝典网站内容还不错，欢迎将脚本宝典推荐好友。

本图文内容来源于网友网络收集整理提供，作为学习参考使用，版权属于原作者。
如您有任何意见或建议可联系处理。小编QQ：384754419，请注明来意。
标签：hadoophdfsjavalinuxlistmapreducepartitionphppythonshuffleStringtcptcpvimvimzookeeper安装安装程序员程序员
猜你在找的心得技巧相关文章
其他相关热搜词更多
phpjavapython程序员loadpost-format-gallerydivString参数开发ListcapMapClass工具安装SQL数据库资源this
全站导航更多
html5HTML/XhtmlCSSXML/XSLTDreamweaver教程Frontpage教程心得技巧JavaScriptASP.NETPHP编程正则表达式AJAX相关ASP编程JSP编程编程10000问CSS/HTMLFlexvbsDOS/BAThtahtcpythonperl游戏相关VBA远程脚本ColdFusionMsSqlMysqlmariadboracleDB2mssql2008mssql2005SQLitePostgreSQLMongoDB星外虚拟主机华众虚拟主机Linuxwin服务器FTP服务器DNS服务器Tomcatnginxzabbix云和虚拟化bios系统安装系统系统进程Windows系列LINUXRedHat/CentosUbuntu/DebianFedoraSolaris红旗Linux建站经验微信营销网站优化网站策划网络赚钱网络创业站长故事alexa域名photoshop教程摄影教程Fireworks教程CorelDraw教程Illustrator教程Painter教程Freehand教程IndesignSketch笔记本主板内存CPU存储显卡显示器光存储鼠标键盘平板电脑安全教程杀毒防毒安全设置病毒查杀脚本攻防入侵防御工具使用业界动态Exploit漏洞分析
最新心得技巧教程
热门心得技巧教程