大数据-消息队列：Kafka【分布式的基于push/pull模式的消息队列(Message Queue)；在流式计算中，Kafka一般用来缓存数据，Spark通过消费Kafka的数据进行计算】

向右看齐 2023-10-05 17:50 34阅读 0赞

![在这里插入图片描述][20210125191658808.png]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTMyNTA4NjE_size_16_color_FFFFFF_t_70_pic_center]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTMyNTA4NjE_size_16_color_FFFFFF_t_70_pic_center 1]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTMyNTA4NjE_size_16_color_FFFFFF_t_70_pic_center 2]

## 一、Kafka集群的安装 ##

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTMyNTA4NjE_size_16_color_FFFFFF_t_70_pic_center 3]

### 1、在hadoop102服务器上安装Kafka ###

解压安装包，其中的2.11表示Kafka的Scala语言版本，0.11.0表示Kafka版本

[wyr@hadoop102 soft]$ tar -zxvf kafka_2.11-0.11.0.0.tgz  -C ../module/

修改解压后的文件名称

drwxrwxr-x.  9 wyr wyr 4096 Jan 31 14:45 flume
    drwxr-xr-x. 11 wyr wyr 4096 Jan 31 10:43 hadoop-2.7.2
    drwxrwxr-x.  9 wyr wyr 4096 Jan 30 19:27 hive
    drwxr-xr-x.  8 wyr wyr 4096 Dec 13  2016 jdk1.8.0_121
    drwxr-xr-x.  6 wyr wyr 4096 Jun 23  2017 kafka_2.11-0.11.0.0
    drwxr-xr-x. 11 wyr wyr 4096 Jan 29 22:01 zookeeper-3.4.10
    [wyr@hadoop102 module]$ mv kafka_2.11-0.11.0.0/ kafka
    [wyr@hadoop102 module]$ ll
    total 24
    drwxrwxr-x.  9 wyr wyr 4096 Jan 31 14:45 flume
    drwxr-xr-x. 11 wyr wyr 4096 Jan 31 10:43 hadoop-2.7.2
    drwxrwxr-x.  9 wyr wyr 4096 Jan 30 19:27 hive
    drwxr-xr-x.  8 wyr wyr 4096 Dec 13  2016 jdk1.8.0_121
    drwxr-xr-x.  6 wyr wyr 4096 Jun 23  2017 kafka
    drwxr-xr-x. 11 wyr wyr 4096 Jan 29 22:01 zookeeper-3.4.10
    [wyr@hadoop102 module]$

### 2、配置 ###

#### 2.1 环境变量 ####

1.  Kafka使用Scala语言编写的，所以要保证安装Kafka的Linux服务器安装JDK，环境变量中有JAVA\_HOME
2.  Kafka会向Zookeeper中存储一些元数据，因此需要安装Zookeeper

#### 2.2 修改配置文件 ####

在/opt/module/kafka目录下创建datas文件夹(数据文件的存放目录)

[wyr@hadoop102 kafka]$ mkdir datas
    [wyr@hadoop102 kafka]$ ll
    total 56
    drwxr-xr-x. 3 wyr wyr  4096 Jun 23  2017 bin
    drwxr-xr-x. 2 wyr wyr  4096 Jun 23  2017 config
    drwxrwxr-x. 2 wyr wyr  4096 Feb  1 16:12 datas
    drwxr-xr-x. 2 wyr wyr  4096 Feb  1 15:57 libs
    -rw-r--r--. 1 wyr wyr 28824 Jun 23  2017 LICENSE
    -rw-r--r--. 1 wyr wyr   336 Jun 23  2017 NOTICE
    drwxr-xr-x. 2 wyr wyr  4096 Jun 23  2017 site-docs
    [wyr@hadoop102 kafka]$

编辑Kafka安装目录下的 /opt/module/kafka/config/server.properties 文件

#21行，cluster中的每台broker都需要有唯一的id号，必须为整数
    broker.id=102
    #24行，打开注释,此行代表允许手动删除kafka中的主题
    delete.topic.enable=true
    #63行，配置kafka存储的数据文件的存放目录
    log.dirs=/opt/module/kafka/datas
    #126行，配置连接的zk集群的地址
    zookeeper.connect=hadoop101:2181,hadoop102:2181,hadoop103:2181

#### 2.3 将hadoop102上的Kafka目录分发到hadoop101、hadoop103 ####

[wyr@hadoop102 module]$ xsync.sh kafka

分别在hadoop101和hadoop103上修改配置文件/opt/module/kafka/config/server.properties中的broker.id=101、broker.id=103（注：broker.id不得重复）

### 3、启动/停止Kafka集群 ###

Kafka依赖于Zookeeper，所以启动Kafka之前先启动Zookeeper，Kafka可以借助Zookeeper来动态地检测各个broker的上线与下线

#### 3.1 在hadoop102中使用群起脚本群起Zookeeper集群 ####

[wyr@hadoop102 ~]$ xcall.sh /opt/module/zookeeper-3.4.10/bin/zkServer.sh start
    要执行的命令是/opt/module/zookeeper-3.4.10/bin/zkServer.sh start
    ----------------------------hadoop101----------------------------------
    ZooKeeper JMX enabled by default
    Using config: /opt/module/zookeeper-3.4.10/bin/../conf/zoo.cfg
    Starting zookeeper ... STARTED
    ----------------------------hadoop102----------------------------------
    ZooKeeper JMX enabled by default
    Using config: /opt/module/zookeeper-3.4.10/bin/../conf/zoo.cfg
    Starting zookeeper ... STARTED
    ----------------------------hadoop103----------------------------------
    ZooKeeper JMX enabled by default
    Using config: /opt/module/zookeeper-3.4.10/bin/../conf/zoo.cfg
    Starting zookeeper ... STARTED
    [wyr@hadoop102 ~]$ xcall.sh /opt/module/zookeeper-3.4.10/bin/zkServer.sh status
    要执行的命令是/opt/module/zookeeper-3.4.10/bin/zkServer.sh status
    ----------------------------hadoop101----------------------------------
    ZooKeeper JMX enabled by default
    Using config: /opt/module/zookeeper-3.4.10/bin/../conf/zoo.cfg
    Mode: leader
    ----------------------------hadoop102----------------------------------
    ZooKeeper JMX enabled by default
    Using config: /opt/module/zookeeper-3.4.10/bin/../conf/zoo.cfg
    Mode: follower
    ----------------------------hadoop103----------------------------------
    ZooKeeper JMX enabled by default
    Using config: /opt/module/zookeeper-3.4.10/bin/../conf/zoo.cfg
    Mode: follower
    [wyr@hadoop102 ~]$ xcall.sh jps
    要执行的命令是jps
    ----------------------------hadoop101----------------------------------
    4181 QuorumPeerMain
    4317 Jps
    ----------------------------hadoop102----------------------------------
    4657 Jps
    4515 QuorumPeerMain
    ----------------------------hadoop103----------------------------------
    3804 QuorumPeerMain
    3934 Jps
    [wyr@hadoop102 ~]$

在hadoop101上登陆Zookeeper的客户端

[wyr@hadoop101 ~]$ /opt/module/zookeeper-3.4.10/bin/zkCli.sh -server hadoop101:2181
    Connecting to hadoop101:2181
    2021-02-01 16:29:24,726 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
    2021-02-01 16:29:24,728 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=hadoop101
    2021-02-01 16:29:24,728 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_121
    2021-02-01 16:29:24,729 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
    2021-02-01 16:29:24,729

[20210125191658808.png]: https://img-blog.csdnimg.cn/20210125191658808.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTMyNTA4NjE_size_16_color_FFFFFF_t_70_pic_center]: https://img-blog.csdnimg.cn/20210124133315504.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTMyNTA4NjE=,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTMyNTA4NjE_size_16_color_FFFFFF_t_70_pic_center 1]: https://img-blog.csdnimg.cn/20210606110848771.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTMyNTA4NjE=,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTMyNTA4NjE_size_16_color_FFFFFF_t_70_pic_center 2]: https://img-blog.csdnimg.cn/20210606111053877.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTMyNTA4NjE=,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTMyNTA4NjE_size_16_color_FFFFFF_t_70_pic_center 3]: https://img-blog.csdnimg.cn/20210606105022668.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTMyNTA4NjE=,size_16,color_FFFFFF,t_70#pic_center