Kafka服务器配置的offsets.retention.minutes & log.retention.minutes默认参数配置导致offset失效的问题

快来打我* 2021-11-26 08:34 405阅读 0赞

关于kafka 0.10.0.0 的默认参数 offsets.retention.minutes & log.retention.minutes 的默认值问题。

默认参数前者是7天，后者是24小时。会导致数据虽然保存但offset失效导致客户端数据重复消费的问题。

0.10.0.0官方的参数说明：[http://kafka.apache.org/0100/documentation.html\#log][http_kafka.apache.org_0100_documentation.html_log]

offsets.retention.minutes

Log retention window in minutes for offsets topic

Kafka Server端保存的offset的过期时间。默认值1440（1440分钟也就是24小时），应该调整为与log.retention.hours一致，即10080。

log.retention.hours & log.retention.minutes

这两个参数都是用来设置删除日志的，无论哪个属性已经溢出，都会进行文件的删除。

log.retention.hours:   
The number of hours to keep a log file before deleting it (in hours), tertiary to log.retention.ms property  
参数int类型，默认值：168 （168小时也就是7天）。

log.retention.minutes:   
The number of minutes to keep a log file before deleting it (in minutes), secondary to log.retention.ms property. If not set, the value in log.retention.hours is used  
参数int类型，默认值：null。

该问题相关的问题单和解决方案：默认值调成一致(7天)：

类似相关问题单很多：

[https://cwiki.apache.org/confluence/display/KAFKA/KIP-186%3A+Increase+offsets+retention+default+to+7+days][https_cwiki.apache.org_confluence_display_KAFKA_KIP-186_3A_Increase_offsets_retention_default_to_7_days]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3N1bm55MDUyOTY_size_16_color_FFFFFF_t_70][]

[https://github.com/apache/kafka/pull/4648/files][https_github.com_apache_kafka_pull_4648_files]

# KAFKA-3806: Increase offsets retention default to 7 days (KIP-186) \#4648 #

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3N1bm55MDUyOTY_size_16_color_FFFFFF_t_70 1][]

[http_kafka.apache.org_0100_documentation.html_log]: http://kafka.apache.org/0100/documentation.html#log
[https_cwiki.apache.org_confluence_display_KAFKA_KIP-186_3A_Increase_offsets_retention_default_to_7_days]: https://cwiki.apache.org/confluence/display/KAFKA/KIP-186%3A+Increase+offsets+retention+default+to+7+days
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3N1bm55MDUyOTY_size_16_color_FFFFFF_t_70]: /images/20211126/8d9e3fecfdb84d8db5c0c5b4ee6c26d0.png
[https_github.com_apache_kafka_pull_4648_files]: https://github.com/apache/kafka/pull/4648/files
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3N1bm55MDUyOTY_size_16_color_FFFFFF_t_70 1]: /images/20211126/8ae056450146474382e3a5a2fef71afa.png