【Kafka】记录一次虚拟机环境Kafka benchmark测试

梦里梦外; 2023-10-05 16:41 34阅读 0赞

## 注意 ##

*  此性能测试在虚拟机上测试！
 *  kafka基准测试，消息发送，消息消费
 *  集群规模为三个broker
 *  此测试对topic不同分区进行测试
 *  2.1版本kafka已经取消了多少线程发送的测试，所以不对发送线程进行归类测试

## 集群环境 ##

**机器：**测试机器163，164，165

**kafka Info：**3节点broker，jvm 1g，日志168小时(测试时，单独对topic进行了日志存储时间的设置，为的是自动清除数据，测试机器硬盘小)

**kafka version：**2.1.1

**虚拟机配置：**8 Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz 8个逻辑核 64位

1 physical id : 0  
1 address sizes : 43 bits physical, 48 bits virtual  
1 physical id : 2  
1 address sizes : 43 bits physical, 48 bits virtual  
1 physical id : 4  
1 address sizes : 43 bits physical, 48 bits virtual  
1 physical id : 6  
1 address sizes : 43 bits physical, 48 bits virtual  
1 physical id : 8  
1 address sizes : 43 bits physical, 48 bits virtual  
1 physical id : 10  
1 address sizes : 43 bits physical, 48 bits virtual  
1 physical id : 12  
1 address sizes : 43 bits physical, 48 bits virtual  
1 physical id : 14  
1 address sizes : 43 bits physical, 48 bits virtual

虚拟机核不做过多赘述

Linux XXGL-T-TJSYZ-app-026 3.10.0-1062.12.1.el7.x86\_64 \#1 SMP Tue Feb 4 23:02:59 UTC 2020 x86\_64 x86\_64 x86\_64 GNU/Linux

## 测试环境 ##

**机器：**172.24.29.163（虚拟机）

**虚拟机配置：**8 Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz

## 性能测试工具 ##

kafka自带测试脚本：kafka-producer-perf-test.sh、kafka-consumer-perf-test.sh

## 生产者测试 ##

### 分区数不同 ###

分区数分别为3、9、15，消息数都为：100W，消息大小：100byte，副本数都为1

**创建Topic：**

注意：topic名称之前不能有空格

./kafka-topics.sh -create -zookeeper 172.24.29.163:2181/kafka211 -topic test\_producer\_3\_1 -partitions 3 -replication-factor 1

./kafka-topics.sh -create -zookeeper 172.24.29.163:2181/kafka211 -topic test\_producer\_6\_1 -partitions 6 -replication-factor 1

./kafka-topics.sh -create -zookeeper 172.24.29.163:2181/kafka211 -topic test\_producer\_9\_1 -partitions 9 -replication-factor 1

./kafka-topics.sh -create -zookeeper 172.24.29.163:2181/kafka211 -topic test\_producer\_12\_1 -partitions 12 -replication-factor 1

./kafka-topics.sh -create -zookeeper 172.24.29.163:2181/kafka211 -topic test\_producer\_15\_1 -partitions 15 -replication-factor 1

**设置topic消息保存时间：**

修改消息保存时间为1分钟

./kafka-configs.sh --zookeeper 172.24.29.163:2181/kafka211 --entity-type topics --entity-name test\_producer\_3\_1 --add-config [retention.ms][]=60000 --alter

./kafka-configs.sh --zookeeper 172.24.29.163:2181/kafka211 --entity-type topics --entity-name test\_producer\_9\_1 --add-config [retention.ms][]=60000 --alter

./kafka-configs.sh --zookeeper 172.24.29.163:2181/kafka211 --entity-type topics --entity-name test\_producer\_15\_1 --add-config [retention.ms][]=60000 --alter

**生产消息：**

./kafka-producer-perf-test.sh --num-records 1000000 --topic test\_producer\_3\_1 --throughput 100000000 --record-size 100 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1

1000000 records sent, 811688.311688 records/sec (77.41 MB/sec), 5.51 ms avg latency, 228.00 ms max latency, 5 ms 50th, 12 ms 95th, 15 ms 99th, 29 ms 99.9th.

./kafka-producer-perf-test.sh --num-records 1000000 --topic test\_producer\_6\_1 --throughput 100000000 --record-size 100 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1

1000000 records sent, 827129.859388 records/sec (78.88 MB/sec), 5.05 ms avg latency, 180.00 ms max latency, 3 ms 50th, 13 ms 95th, 17 ms 99th, 23 ms 99.9th.

./kafka-producer-perf-test.sh --num-records 1000000 --topic test\_producer\_9\_1 --throughput 100000000 --record-size 100 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1

1000000 records sent, 840336.134454 records/sec (80.14 MB/sec), 6.88 ms avg latency, 169.00 ms max latency, 4 ms 50th, 16 ms 95th, 32 ms 99th, 35 ms 99.9th.

./kafka-producer-perf-test.sh --num-records 1000000 --topic test\_producer\_12 --throughput 100000000 --record-size 100 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1

1000000 records sent, 769230.769231 records/sec (73.36 MB/sec), 6.08 ms avg latency, 184.00 ms max latency, 4 ms 50th, 18 ms 95th, 43 ms 99th, 66 ms 99.9th.

./kafka-producer-perf-test.sh --num-records 1000000 --topic test\_producer\_15\_1 --throughput 100000000 --record-size 100 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1

1000000 records sent, 847457.627119 records/sec (80.82 MB/sec), 5.38 ms avg latency, 170.00 ms max latency, 4 ms 50th, 12 ms 95th, 21 ms 99th, 32 ms 99.9th.

**结果：**

<table> 
 <tbody> 
  <tr> 
   <th>节点数</th> 
   <th>分区数</th> 
   <th>副本数</th> 
   <th> <p>每秒发送消息数&nbsp; records/sec</p> </th> 
   <th>发送速率&nbsp;MB/sec</th> 
   <th>平均延迟</th> 
   <th>最大延迟</th> 
   <th>50%</th> 
   <th>95%</th> 
   <th>99%</th> 
  </tr> 
  <tr> 
   <td>3</td> 
   <td>3</td> 
   <td>1</td> 
   <td>811688</td> 
   <td>77.41&nbsp;</td> 
   <td>5.51 ms</td> 
   <td>228.00 ms</td> 
   <td>5ms</td> 
   <td>12ms</td> 
   <td>15ms</td> 
  </tr> 
  <tr> 
   <td>3</td> 
   <td>6</td> 
   <td>1</td> 
   <td>827130</td> 
   <td>78.88</td> 
   <td>5.05 ms</td> 
   <td>180.00 ms</td> 
   <td>3ms</td> 
   <td>13ms</td> 
   <td>17ms</td> 
  </tr> 
  <tr> 
   <td>3</td> 
   <td>9</td> 
   <td>1</td> 
   <td>840336</td> 
   <td>80.14</td> 
   <td>6.88 ms</td> 
   <td>169.00 ms</td> 
   <td>4ms</td> 
   <td>16ms</td> 
   <td>32ms</td> 
  </tr> 
  <tr> 
   <td>3</td> 
   <td>12</td> 
   <td>1</td> 
   <td>769231</td> 
   <td>73.36</td> 
   <td>6.08 ms</td> 
   <td>184.00 ms</td> 
   <td>4ms</td> 
   <td>18ms</td> 
   <td>43ms</td> 
  </tr> 
  <tr> 
   <td>3</td> 
   <td>15</td> 
   <td>1</td> 
   <td>847458</td> 
   <td>80.82</td> 
   <td>5.38 ms</td> 
   <td>170.00 ms</td> 
   <td>4ms</td> 
   <td>12ms</td> 
   <td>21ms</td> 
  </tr> 
 </tbody> 
</table>

### 副本数不同 ###

分区数都为9，消息数都为：100W，消息大小：100byte，副本数分别为1、2、3

**创建Topic：**

./kafka-topics.sh -create -zookeeper 172.24.29.163:2181/kafka211 -topic test\_producer\_9\_2 -partitions 9 -replication-factor 2

./kafka-topics.sh -create -zookeeper 172.24.29.163:2181/kafka211 -topic test-producer-9-3 -partitions 9 -replication-factor 3

**设置topic消息保存时间：**

./kafka-configs.sh --zookeeper 172.24.29.163:2181/kafka211 --entity-type topics --entity-name test\_producer\_9\_2 --add-config [retention.ms][]=60000 --alter

./kafka-configs.sh --zookeeper 172.24.29.163:2181/kafka211 --entity-type topics --entity-name test\_producer\_9\_3 --add-config [retention.ms][]=60000 --alter

**生产消息：**

./kafka-producer-perf-test.sh --num-records 1000000 --topic test\_producer\_9\_1 --throughput 100000000 --record-size 100 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1

1000000 records sent, 803858.520900 records/sec (76.66 MB/sec), 3.00 ms avg latency, 194.00 ms max latency, 3 ms 50th, 6 ms 95th, 10 ms 99th, 12 ms 99.9th.

./kafka-producer-perf-test.sh --num-records 1000000 --topic test\_producer\_9\_2 --throughput 100000000 --record-size 100 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1

1000000 records sent, 803212.851406 records/sec (76.60 MB/sec), 7.69 ms avg latency, 183.00 ms max latency, 6 ms 50th, 19 ms 95th, 23 ms 99th, 24 ms 99.9th.

./kafka-producer-perf-test.sh --num-records 1000000 --topic test\_producer\_9\_3 --throughput 100000000 --record-size 100 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1

1000000 records sent, 795544.948290 records/sec (75.87 MB/sec), 13.01 ms avg latency, 194.00 ms max latency, 11 ms 50th, 31 ms 95th, 45 ms 99th, 56 ms 99.9th.

结果：

<table> 
 <tbody> 
  <tr> 
   <th>节点数</th> 
   <th>分区数</th> 
   <th>副本数</th> 
   <th> <p>每秒发送消息数&nbsp; records/sec</p> </th> 
   <th>发送速率&nbsp;MB/sec</th> 
   <th>平均延迟 ms</th> 
   <th>最大延迟 ms</th> 
   <th>50% ms</th> 
   <th>95% ms</th> 
   <th>99% ms</th> 
   <th>99.9% ms</th> 
  </tr> 
  <tr> 
   <td>3</td> 
   <td>9</td> 
   <td>1</td> 
   <td>803859</td> 
   <td>76.66</td> 
   <td>3.00</td> 
   <td>194</td> 
   <td>3</td> 
   <td>6</td> 
   <td>10</td> 
   <td>12</td> 
  </tr> 
  <tr> 
   <td>3</td> 
   <td>9</td> 
   <td>2</td> 
   <td>803213</td> 
   <td>76.60</td> 
   <td>7.69</td> 
   <td>183</td> 
   <td>6</td> 
   <td>19</td> 
   <td>23</td> 
   <td>24</td> 
  </tr> 
  <tr> 
   <td>3</td> 
   <td>9</td> 
   <td>3</td> 
   <td>803213</td> 
   <td>75.87</td> 
   <td>13.01</td> 
   <td>194</td> 
   <td>11</td> 
   <td>31</td> 
   <td>45</td> 
   <td>56</td> 
  </tr> 
 </tbody> 
</table>

### 消息大小不同 ###

分区数都为9，消息数都为：100W，副本数分别为1，消息大小分别为：10byte、100byte、1000byte

**生产消息：**

./kafka-producer-perf-test.sh --num-records 1000000 --topic test\_producer\_9\_1 --throughput 100000000 --record-size 10 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1

1000000 records sent, 869565.217391 records/sec (8.29 MB/sec), 3.12 ms avg latency, 180.00 ms max latency, 3 ms 50th, 6 ms 95th, 16 ms 99th, 22 ms 99.9th.

./kafka-producer-perf-test.sh --num-records 1000000 --topic test\_producer\_9\_1 --throughput 100000000 --record-size 100 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1

1000000 records sent, 812347.684809 records/sec (77.47 MB/sec), 4.67 ms avg latency, 169.00 ms max latency, 3 ms 50th, 12 ms 95th, 13 ms 99th, 21 ms 99.9th.

./kafka-producer-perf-test.sh --num-records 1000000 --topic test\_producer\_9\_1 --throughput 100000000 --record-size 1000 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1

1000000 records sent, 233972.859148 records/sec (223.13 MB/sec), 124.23 ms avg latency, 282.00 ms max latency, 150 ms 50th, 238 ms 95th, 260 ms 99th, 280 ms 99.9th.

**结果：**

<table> 
 <tbody> 
  <tr> 
   <th>节点数</th> 
   <th>分区数</th> 
   <th>副本数</th> 
   <th>消息大小 byte</th> 
   <th> <p>每秒发送消息数&nbsp; records/sec</p> </th> 
   <th>发送速率&nbsp;MB/sec</th> 
   <th>平均延迟 ms</th> 
   <th>最大延迟 ms</th> 
   <th>50% ms</th> 
   <th>95% ms</th> 
   <th>99% ms</th> 
   <th>99.9% ms</th> 
  </tr> 
  <tr> 
   <td>3</td> 
   <td>9</td> 
   <td>1</td> 
   <td>10</td> 
   <td>869565</td> 
   <td>8.29</td> 
   <td>3.12</td> 
   <td>180</td> 
   <td>3</td> 
   <td>6</td> 
   <td>16</td> 
   <td>22</td> 
  </tr> 
  <tr> 
   <td>3</td> 
   <td>9</td> 
   <td>1</td> 
   <td>100</td> 
   <td>812348</td> 
   <td>77.47</td> 
   <td>4.67</td> 
   <td>169</td> 
   <td>3</td> 
   <td>12</td> 
   <td>13</td> 
   <td>21</td> 
  </tr> 
  <tr> 
   <td>3</td> 
   <td>9</td> 
   <td>1</td> 
   <td>1000</td> 
   <td>233973</td> 
   <td>223.13</td> 
   <td>124.23</td> 
   <td>282</td> 
   <td>150</td> 
   <td>283</td> 
   <td>260</td> 
   <td>280</td> 
  </tr> 
 </tbody> 
</table>

## 消费者测试 ##

### 线程数不同 ###

为方便测试，修改消息保存时间为1h：

./kafka-configs.sh --zookeeper 172.24.29.163:2181/kafka211 --entity-type topics --entity-name test\_producer\_3\_1 --add-config [retention.ms][]=600000 --alter

**生产消息：**

节点数：3、分区数：3、副本数：1、消息大小：1000byte

./kafka-producer-perf-test.sh --num-records 1000000 --topic test\_producer\_3\_1 --throughput 100000000 --record-size 1000 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1

**消费消息：**

注意使用不同的消费组消费同一topic

start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec  
2020-10-21 16:56:13:720, 2020-10-21 16:56:15:128, 363.5883, 258.2304, 1000000, 710227.2727, 14, 1394, 260.8238, 717360.1148

start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec  
2020-10-21 16:56:33:331, 2020-10-21 16:56:34:686, 363.9135, 268.5709, 1000341, 738259.0406, 11, 1344, 270.7690, 744301.3393

./kafka-consumer-perf-test.sh --broker-list 172.24.29.163:9092,172.24.29.164:9092,172.24.29.165:9092 --messages 1000000 --fetch-size 1048576 --topic test\_producer\_3\_1 --group test-group-3 --timeout 60000 --threads 20  
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec  
2020-10-21 16:56:48:238, 2020-10-21 16:56:49:648, 363.5883, 257.8641, 1000000, 709219.8582, 13, 1397, 260.2637, 715819.6135

start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec  
2020-10-21 16:57:06:195, 2020-10-21 16:57:07:610, 363.5883, 256.9529, 1000000, 706713.7809, 12, 1403, 259.1506, 712758.3749

start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec  
2020-10-21 16:58:05:760, 2020-10-21 16:58:07:201, 363.9708, 252.5821, 1000401, 694240.8050, 15, 1426, 255.2390, 701543.4783

**解释：**

start.time：测试开始时间,

end.time：测试结束时间,

data.consumed.in.MB：消费的所有消息的大小(MB),

MB.sec：每秒传输的数据量(MB),

data.consumed.in.nMsg：测试总共消费的消息数,

nMsg.sec：一秒内消费的消息数,

rebalance.time.ms：均衡耗时(ms),

fetch.time.ms：拉取耗时(ms),

fetch.MB.sec：每秒拉取数据(MB),

fetch.nMsg.sec：每秒拉取消息数

**结果：**

<table> 
 <tbody> 
  <tr> 
   <th>线程数</th> 
   <th> <p>消息总大小(MB)</p> <p>data.consumed.in.MB</p> </th> 
   <th> <p>每秒传输数据大小(MB)</p> <p>MB.sec</p> </th> 
   <th> <p>消费消息总数</p> <p>data.consumed.in.nMsg</p> </th> 
   <th> <p>每秒消费消息数</p> <p>nMsg.sec</p> </th> 
   <th> <p>均衡耗时(ms)</p> <p>rebalance.time.ms</p> </th> 
   <th> <p>拉取耗时(ms)</p> <p>fetch.time.ms</p> </th> 
   <th> <p>每秒拉取数据(MB)</p> <p>fetch.MB.sec</p> </th> 
   <th> <p>每秒拉取消息数</p> <p>fetch.nMsg.sec</p> </th> 
  </tr> 
  <tr> 
   <td>1</td> 
   <td>363.5883</td> 
   <td>258.2304</td> 
   <td>1000000</td> 
   <td>710227.2727</td> 
   <td>14</td> 
   <td>1394</td> 
   <td>260.8238</td> 
   <td>717360.1148</td> 
  </tr> 
  <tr> 
   <td>10</td> 
   <td>363.5883</td> 
   <td>268.5709</td> 
   <td>1000000</td> 
   <td>738259.0406</td> 
   <td>11</td> 
   <td>1344</td> 
   <td>270.7690</td> 
   <td>744301.3393</td> 
  </tr> 
  <tr> 
   <td>20</td> 
   <td>363.5883</td> 
   <td>257.8641</td> 
   <td>1000000</td> 
   <td>709219.8582</td> 
   <td>13</td> 
   <td>1397</td> 
   <td>260.2637</td> 
   <td>715819.6135</td> 
  </tr> 
  <tr> 
   <td>30</td> 
   <td>363.5883</td> 
   <td>256.9529</td> 
   <td>1000000</td> 
   <td>706713.7809</td> 
   <td>12</td> 
   <td>1403</td> 
   <td>259.1506</td> 
   <td>712758.3749</td> 
  </tr> 
  <tr> 
   <td>40</td> 
   <td>363.5883</td> 
   <td>252.5821</td> 
   <td>1000000</td> 
   <td>694240.8050</td> 
   <td>15</td> 
   <td>1426</td> 
   <td>255.2390</td> 
   <td>701543.4783</td> 
  </tr> 
 </tbody> 
</table>

### 分区数不同 ###

**生产消息：**

消息大小均为100byte、消息数：100W、节点数：3、副本数：1

./kafka-producer-perf-test.sh --num-records 1000000 --topic test\_producer\_3\_1 --throughput 100000000 --record-size 100 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1

./kafka-producer-perf-test.sh --num-records 1000000 --topic test\_producer\_6\_1 --throughput 100000000 --record-size 100 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1

./kafka-producer-perf-test.sh --num-records 1000000 --topic test\_producer\_9\_1 --throughput 100000000 --record-size 100 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1

./kafka-producer-perf-test.sh --num-records 1000000 --topic test\_producer\_12\_1 --throughput 100000000 --record-size 100 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1

./kafka-producer-perf-test.sh --num-records 1000000 --topic test\_producer\_15\_1 --throughput 100000000 --record-size 100 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1

**消费消息：**

start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec  
2020-10-21 17:36:08:761, 2020-10-21 17:36:09:683, 95.3674, 103.4354, 1000000, 1084598.6985, 11, 911, 104.6843, 1097694.8408

start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec  
2020-10-21 17:35:12:812, 2020-10-21 17:35:13:746, 95.3674, 102.1065, 1000000, 1070663.8116, 12, 922, 103.4354, 1084598.6985

start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec  
2020-10-21 17:37:00:262, 2020-10-21 17:37:01:155, 95.3674, 106.7944, 1000000, 1119820.8287, 13, 880, 108.3721, 1136363.6364

start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec  
2020-10-21 17:37:45:271, 2020-10-21 17:37:46:176, 95.3674, 105.3784, 1000000, 1104972.3757, 12, 893, 106.7944, 1119820.8287

start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec  
2020-10-21 17:38:29:756, 2020-10-21 17:38:30:659, 95.3674, 105.6118, 1000000, 1107419.7121, 13, 890, 107.1544, 1123595.5056

**结果：**

<table> 
 <tbody> 
  <tr> 
   <th>分区数</th> 
   <th> <p>消息总大小(MB)</p> <p>data.consumed.in.MB</p> </th> 
   <th> <p>每秒传输数据大小(MB)</p> <p>MB.sec</p> </th> 
   <th> <p>消费消息总数</p> <p>data.consumed.in.nMsg</p> </th> 
   <th> <p>每秒消费消息数</p> <p>nMsg.sec</p> </th> 
   <th> <p>均衡耗时(ms)</p> <p>rebalance.time.ms</p> </th> 
   <th> <p>拉取耗时(ms)</p> <p>fetch.time.ms</p> </th> 
   <th> <p>每秒拉取数据(MB)</p> <p>fetch.MB.sec</p> </th> 
   <th> <p>每秒拉取消息数</p> <p>fetch.nMsg.sec</p> </th> 
  </tr> 
  <tr> 
   <td>3</td> 
   <td>95.3674</td> 
   <td>103.4354</td> 
   <td>1000000</td> 
   <td>1084598.6985</td> 
   <td>11</td> 
   <td>911</td> 
   <td>104.6843</td> 
   <td>1097694.8408</td> 
  </tr> 
  <tr> 
   <td>6</td> 
   <td>95.3674</td> 
   <td>102.1065</td> 
   <td>1000000</td> 
   <td>1070663.8116</td> 
   <td>12</td> 
   <td>922</td> 
   <td>103.4354</td> 
   <td>1084598.6985</td> 
  </tr> 
  <tr> 
   <td>9</td> 
   <td>95.3674</td> 
   <td>106.7944</td> 
   <td>1000000</td> 
   <td>1119820.8287</td> 
   <td>13</td> 
   <td>880</td> 
   <td>108.3721</td> 
   <td>1136363.6364</td> 
  </tr> 
  <tr> 
   <td>12</td> 
   <td>95.3674</td> 
   <td>105.3784</td> 
   <td>1000000</td> 
   <td>1104972.3757</td> 
   <td>12</td> 
   <td>893</td> 
   <td>106.7944</td> 
   <td>1119820.8287</td> 
  </tr> 
  <tr> 
   <td>15</td> 
   <td>95.3674</td> 
   <td>105.6118</td> 
   <td>1000000</td> 
   <td>1107419.7121</td> 
   <td>13</td> 
   <td>890</td> 
   <td>107.1544</td> 
   <td>1123595.5056</td> 
  </tr> 
 </tbody> 
</table>

### 消息大小不同 ###

**生产消息：**

消息数：100W、节点数：3、副本数：1、分区数：3、消息大小分别为：10byte、100byte、1000byte

./kafka-producer-perf-test.sh --num-records 1000000 --topic test\_producer\_3\_1 --throughput 100000000 --record-size 10 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1

./kafka-producer-perf-test.sh --num-records 1000000 --topic test\_producer\_3\_1 --throughput 100000000 --record-size 100 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1

./kafka-producer-perf-test.sh --num-records 1000000 --topic test\_producer\_3\_1 --throughput 100000000 --record-size 1000 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1

**消费消息：**

start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec  
2020-10-21 17:54:21:921, 2020-10-21 17:54:22:634, 9.5367, 13.3755, 1000000, 1402524.5442, 13, 700, 13.6239, 1428571.4286

start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec  
2020-10-21 17:54:56:167, 2020-10-21 17:54:57:050, 95.3674, 108.0039, 1000000, 1132502.8313, 13, 870, 109.6177, 1149425.2874

start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec  
2020-10-21 17:55:25:427, 2020-10-21 17:55:27:890, 953.6743, 387.2003, 1000000, 406008.9322, 19, 2444, 390.2104, 409165.3028

**结果：**

<table> 
 <tbody> 
  <tr> 
   <th> <p>消息大小</p> <p>(bytes)</p> </th> 
   <th> <p>消息总大小(MB)</p> <p>data.consumed.in.MB</p> </th> 
   <th> <p>每秒传输数据大小(MB)</p> <p>MB.sec</p> </th> 
   <th> <p>消费消息总数</p> <p>data.consumed.in.nMsg</p> </th> 
   <th> <p>每秒消费消息数</p> <p>nMsg.sec</p> </th> 
   <th> <p>均衡耗时(ms)</p> <p>rebalance.time.ms</p> </th> 
   <th> <p>拉取耗时(ms)</p> <p>fetch.time.ms</p> </th> 
   <th> <p>每秒拉取数据(MB)</p> <p>fetch.MB.sec</p> </th> 
   <th> <p>每秒拉取消息数</p> <p>fetch.nMsg.sec</p> </th> 
  </tr> 
  <tr> 
   <td>10</td> 
   <td>9.5367</td> 
   <td>13.3755</td> 
   <td>1000000</td> 
   <td>1402524.5442</td> 
   <td>13</td> 
   <td>700</td> 
   <td>13.6239</td> 
   <td>1428571.4286</td> 
  </tr> 
  <tr> 
   <td>100</td> 
   <td>95.3674</td> 
   <td>108.0039</td> 
   <td>1000000</td> 
   <td>1132502.8313</td> 
   <td>13</td> 
   <td>870</td> 
   <td>109.6177</td> 
   <td>1149425.2874</td> 
  </tr> 
  <tr> 
   <td>1000</td> 
   <td>953.6743</td> 
   <td>387.2003</td> 
   <td>1000000</td> 
   <td>406008.9322</td> 
   <td>19</td> 
   <td>2444</td> 
   <td>390.2104</td> 
   <td>409165.3028</td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
  </tr> 
 </tbody> 
</table>

### 每次拉取消息大小不同 ###

**生产消息：**

消息数：100W、每条消息：1000byte、节点数：3、副本数：1、分区数：3、每次请求拉取数据大小分别为：1KB、1MB(默认)、10MB

./kafka-producer-perf-test.sh --num-records 1000000 --topic test\_producer\_3\_1 --throughput 100000000 --record-size 1000 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1

**消费消息：**

./kafka-consumer-perf-test.sh --broker-list 172.24.29.163:9092,172.24.29.164:9092,172.24.29.165:9092 --messages 1000000 --fetch-size 1024 --topic test\_producer\_3\_1 --group test-group-1 --timeout 60000 --threads 10

start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec  
2020-10-21 18:11:25:471, 2020-10-21 18:11:39:242, 953.6743, 69.2524, 1000000, 72616.3677, 12, 13759, 69.3128, 72679.7006

./kafka-consumer-perf-test.sh --broker-list 172.24.29.163:9092,172.24.29.164:9092,172.24.29.165:9092 --messages 1000000 --fetch-size 1024000 --topic test\_producer\_3\_1 --group test-group-2 --timeout 60000 --threads 10

start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec  
2020-10-21 18:11:51:403, 2020-10-21 18:11:53:772, 953.6743, 402.5641, 1000000, 422119.0376, 11, 2358, 404.4420, 424088.2103

./kafka-consumer-perf-test.sh --broker-list 172.24.29.163:9092,172.24.29.164:9092,172.24.29.165:9092 --messages 1000000 --fetch-size 10240000 --topic test\_producer\_3\_1 --group test-group-4 --timeout 60000 --threads 10

start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec

2020-10-21 18:15:11:039, 2020-10-21 18:15:13:808, 953.6743, 344.4111, 1000000, 361141.2062, 13, 2756, 346.0357, 362844.7025

**结果：**

<table> 
 <tbody> 
  <tr> 
   <th>每次拉取大小</th> 
   <th> <p>消息总大小(MB)</p> <p>data.consumed.in.MB</p> </th> 
   <th> <p>每秒传输数据大小(MB)</p> <p>MB.sec</p> </th> 
   <th> <p>消费消息总数</p> <p>data.consumed.in.nMsg</p> </th> 
   <th> <p>每秒消费消息数</p> <p>nMsg.sec</p> </th> 
   <th> <p>均衡耗时(ms)</p> <p>rebalance.time.ms</p> </th> 
   <th> <p>拉取耗时(ms)</p> <p>fetch.time.ms</p> </th> 
   <th> <p>每秒拉取数据(MB)</p> <p>fetch.MB.sec</p> </th> 
   <th> <p>每秒拉取消息数</p> <p>fetch.nMsg.sec</p> </th> 
  </tr> 
  <tr> 
   <td>1KB</td> 
   <td>95.3674</td> 
   <td>69.2524</td> 
   <td>1000000</td> 
   <td>72616.3677</td> 
   <td>12</td> 
   <td>13759</td> 
   <td>69.3128</td> 
   <td>72679.7006</td> 
  </tr> 
  <tr> 
   <td>1MB</td> 
   <td>95.3674</td> 
   <td>402.5641</td> 
   <td>1000000</td> 
   <td>422119.0376</td> 
   <td>11</td> 
   <td>2358</td> 
   <td>404.4420</td> 
   <td>424088.2103</td> 
  </tr> 
  <tr> 
   <td>10MB</td> 
   <td>95.3674</td> 
   <td>344.4111</td> 
   <td>1000000</td> 
   <td>361141.2062</td> 
   <td>13</td> 
   <td>2756</td> 
   <td>346.0357</td> 
   <td>362844.7025</td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
  </tr> 
 </tbody> 
</table>

[retention.ms]: http://retention.ms/