Centos7+Flink1.10+StandaloneHA部署

╰+攻爆jí腚メ 2022-11-13 12:29 193阅读 0赞

### Centos7+Flink1.10+StandaloneHA部署 ###

*  前提
 *  计划
 *  flink搭建配置
 *   *   *  1.下载
         *  2.上传centos7并解压
         *  3.修改配置文件
         *  4.复制到另外2台机器
 *  测试

# 前提 #

1）hadoop3.1.4,搭建可以参考：https://blog.csdn.net/sndayYU/article/details/115037256

2）3台服务器，在1）中已设置免密登录：  
192.168.11.131 hadoop1  
192.168.11.132 hadoop2  
192.168.11.133 hadoop3

2）zookeeper3.4.14（别的版本也可以，并不是太重要）：3台机器都装有

3）openjdk1.8

4）确保flink和hadoop的操作用户相同（不然会连不上hdfs），这里用户都是hadoop

# 计划 #

1）在131安装JobManager，133上安装第二个JobManager；

2）在3台服务器上都安装TaskManager；

3）在131机器挂掉的时候，133的JobManager可以成功启用；

备注：

1）flink集群使用hadoop操作；

# flink搭建配置 #

### 1.下载 ###

*  https://mirrors.tuna.tsinghua.edu.cn/apache/flink/flink-1.10.3/flink-1.10.3-bin-scala\_2.12.tgz
 *  https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-10.0/flink-shaded-hadoop-2-uber-2.8.3-10.0.jar  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NuZGF5WVU_size_16_color_FFFFFF_t_70]  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NuZGF5WVU_size_16_color_FFFFFF_t_70 1]  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NuZGF5WVU_size_16_color_FFFFFF_t_70 2]  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NuZGF5WVU_size_16_color_FFFFFF_t_70 3]

### 2.上传centos7并解压 ###

// 使用hadoop用户，部署到/home/hadoop/目录下
    cd /home/hadoop
    
    // rz上传
    rz
    
    // 解压
    tar zxf flink-1.10.3-bin-scala_2.12.tgz
    
    // flink-shaded-hadoop-2-uber-2.8.3-10.0.jar放入flink的lib
    mv flink-shaded-hadoop-2-uber-2.8.3-10.0.jar flink-1.10.3/lib

### 3.修改配置文件 ###

[hadoop@hadoop1 conf]$ pwd
    /home/hadoop/flink-1.10.3/conf

**1）flink-conf.xml**

# 添加下面内容
    high-availability: zookeeper
    high-availability.storageDir: hdfs:///flink/ha/
    high-availability.zookeeper.quorum: hadoop1:2181,hadoop2:2181,hadoop3:2181

**2）maters**

hadoop1:8081
    hadoop3:8081

**3) slaves**

hadoop1
    hadoop2
    hadoop3

### 4.复制到另外2台机器 ###

scp -r /home/hadoop/flink-1.10.3 hadoop2:/home/hadoop
    scp -r /home/hadoop/flink-1.10.3 hadoop3:/home/hadoop

# 测试 #

// 启动zookeeper：3台机器执行下面命令
    /home/hadoop/zookeeper-3.4.14/bin/zkServer.sh start
    // 验证zookeeper是否正确
    /home/hadoop/zookeeper-3.4.14/bin/zkServer.sh status
    
    // 启动hdfs：仅131机器（hadoop1）上启动。可以jps查看
    /home/hadoop/hadoop-3.1.4/sbin/start-dfs.sh
    [hadoop@hadoop1 data]$ jps
    	21846 QuorumPeerMain
    	22262 DataNode
    	22135 NameNode
    	22587 Jps
    
    // 启动flink：仅131机器执行即可
    /home/hadoop/flink-1.10.3/bin/start-cluster.sh

结果hadoop1:8081和hadoop3:8081都可以访问到flink，如下所示：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NuZGF5WVU_size_16_color_FFFFFF_t_70 4]  
使用root用户kill掉hadoop1机器上的JobManager，结果hadoop1:8081访问不通，而hadoop3:8081还是可以访问的，如下：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NuZGF5WVU_size_16_color_FFFFFF_t_70 5]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NuZGF5WVU_size_16_color_FFFFFF_t_70 6]  
即Standalone高可用 部署成功

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NuZGF5WVU_size_16_color_FFFFFF_t_70]: /images/20221022/c73858935174417181f979400140a1a9.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NuZGF5WVU_size_16_color_FFFFFF_t_70 1]: /images/20221022/7083280f1ac74d5fa9c57f45fc81384e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NuZGF5WVU_size_16_color_FFFFFF_t_70 2]: /images/20221022/dda0a6c9895f4a199e089cad5f2af658.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NuZGF5WVU_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20210327012004197.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NuZGF5WVU=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NuZGF5WVU_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20210327021145346.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NuZGF5WVU=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NuZGF5WVU_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20210327021616718.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NuZGF5WVU=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NuZGF5WVU_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/2021032702163217.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NuZGF5WVU=,size_16,color_FFFFFF,t_70