如何优雅地删除Redis大键

比眉伴天荒 2022-06-03 08:00 229阅读 0赞

# 如何优雅地删除Redis大键 #

转载  2016年08月26日 15:59:49

关于Redis大键(Key)，我们从\[空间复杂性\]和访问它的\[时间复杂度\]两个方面来定义大键。  
前者主要表示Redis键的占用内存大小；后者表示Redis集合数据类型(set/hash/list/sorted set)键，所含有的元素个数。以下两个示例：

> 1个大小200MB的String键(String Object最大512MB)；内存空间角度占用较大  
> 1个包含100000000(1kw)个字段的Hash键，对应访问模式(如hgetall)时间复杂度高

因为内存空间复杂性处理耗时都非常小，测试 del 200MB String键耗时约1毫秒，  
而删除一个含有1kw个字段的Hash键，却会阻塞Redis进程数十秒。所以本文只从时间复杂度分析大的集合类键。删除这种大键的风险，以及怎么优雅地删除。

--------------------

在Redis集群中，应用程序尽量避免使用大键；直接影响容易导致集群的容量和请求出现”倾斜问题“，具体分析见文章：[redis-cluster-imbalance][]。但在实际生产过程中，总会有业务使用不合理，出现这类大键；当DBA发现后推进业务优化改造，然后删除这个大键；如果直接删除它，DEL命令可能阻塞Redis进程数十秒，对应用程序和Redis集群可用性造成严重的影响。

## 直接删除大Key的风险 ##

[DEL命令][DEL]在删除单个集合类型的Key时，命令的时间复杂度是O(M)，其中M是集合类型Key包含的元素个数。

> DEL key  
> Time complexity: O(N) where N is the number of keys that will be removed. When a key to remove holds a value other than a string, the individual complexity for this key is O(M) where M is the number of elements in the list, set, sorted set or hash. Removing a single key that holds a string value is O(1).

生产环境中遇到过多次因业务删除大Key，导致Redis阻塞，出现故障切换和应用程序雪崩的故障。  
测试删除集合类型大Key耗时，一般每秒可清理100w~数百w个元素; 如果数千w个元素的大Key时，会导致Redis阻塞上10秒  
可能导致集群判断Redis已经故障，出现故障切换；或应用程序出现雪崩的情况。

> 说明：Redis是单线程处理。  
> 单个耗时过大命令，导致阻塞其他命令，容易引起应用程序雪崩或Redis集群发生故障切换。  
> 所以避免在生产环境中使用耗时过大命令。

Redis删除大的集合键的耗时, 测试估算，可参考；和硬件环境、Redis版本和负载等因素有关

<table style="border-collapse:collapse; border-spacing:0px; margin:20px 0px; border:1px solid rgb(221,221,221); font-size:14px; table-layout:fixed; color:rgb(85,85,85); font-family:Lato,'PingFang SC','Microsoft YaHei',sans-serif; text-align:justify"> 
 <thead> 
  <tr> 
   <th style="padding:8px 8px 10px; text-align:left; vertical-align:middle; border-bottom-width:3px; border-bottom-style:solid; border-bottom-color:rgb(221,221,221); border-right-width:1px; border-right-style:solid; border-right-color:rgb(238,238,238)"> Key类型</th> 
   <th style="padding:8px 8px 10px; text-align:left; vertical-align:middle; border-bottom-width:3px; border-bottom-style:solid; border-bottom-color:rgb(221,221,221); border-right-width:1px; border-right-style:solid; border-right-color:rgb(238,238,238)"> Item数量</th> 
   <th style="padding:8px 8px 10px; text-align:left; vertical-align:middle; border-bottom-width:3px; border-bottom-style:solid; border-bottom-color:rgb(221,221,221); border-right-width:1px; border-right-style:solid; border-right-color:rgb(238,238,238)"> 耗时</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr style="background-color:rgb(249,249,249)"> 
   <td style="padding:8px; text-align:left; vertical-align:middle; border-bottom-width:1px; border-bottom-style:solid; border-bottom-color:rgb(221,221,221); border-right-width:1px; border-right-style:solid; border-right-color:rgb(238,238,238)"> Hash</td> 
   <td style="padding:8px; text-align:left; vertical-align:middle; border-bottom-width:1px; border-bottom-style:solid; border-bottom-color:rgb(221,221,221); border-right-width:1px; border-right-style:solid; border-right-color:rgb(238,238,238)"> ~100万</td> 
   <td style="padding:8px; text-align:left; vertical-align:middle; border-bottom-width:1px; border-bottom-style:solid; border-bottom-color:rgb(221,221,221); border-right-width:1px; border-right-style:solid; border-right-color:rgb(238,238,238)"> ~1000ms</td> 
  </tr> 
  <tr> 
   <td style="padding:8px; text-align:left; vertical-align:middle; border-bottom-width:1px; border-bottom-style:solid; border-bottom-color:rgb(221,221,221); border-right-width:1px; border-right-style:solid; border-right-color:rgb(238,238,238)"> List</td> 
   <td style="padding:8px; text-align:left; vertical-align:middle; border-bottom-width:1px; border-bottom-style:solid; border-bottom-color:rgb(221,221,221); border-right-width:1px; border-right-style:solid; border-right-color:rgb(238,238,238)"> ~100万</td> 
   <td style="padding:8px; text-align:left; vertical-align:middle; border-bottom-width:1px; border-bottom-style:solid; border-bottom-color:rgb(221,221,221); border-right-width:1px; border-right-style:solid; border-right-color:rgb(238,238,238)"> ~1000ms</td> 
  </tr> 
  <tr style="background-color:rgb(249,249,249)"> 
   <td style="padding:8px; text-align:left; vertical-align:middle; border-bottom-width:1px; border-bottom-style:solid; border-bottom-color:rgb(221,221,221); border-right-width:1px; border-right-style:solid; border-right-color:rgb(238,238,238)"> Set</td> 
   <td style="padding:8px; text-align:left; vertical-align:middle; border-bottom-width:1px; border-bottom-style:solid; border-bottom-color:rgb(221,221,221); border-right-width:1px; border-right-style:solid; border-right-color:rgb(238,238,238)"> ~100万</td> 
   <td style="padding:8px; text-align:left; vertical-align:middle; border-bottom-width:1px; border-bottom-style:solid; border-bottom-color:rgb(221,221,221); border-right-width:1px; border-right-style:solid; border-right-color:rgb(238,238,238)"> ~1000ms</td> 
  </tr> 
  <tr> 
   <td style="padding:8px; text-align:left; vertical-align:middle; border-bottom-width:1px; border-bottom-style:solid; border-bottom-color:rgb(221,221,221); border-right-width:1px; border-right-style:solid; border-right-color:rgb(238,238,238)"> Sorted Set</td> 
   <td style="padding:8px; text-align:left; vertical-align:middle; border-bottom-width:1px; border-bottom-style:solid; border-bottom-color:rgb(221,221,221); border-right-width:1px; border-right-style:solid; border-right-color:rgb(238,238,238)"> ~100万</td> 
   <td style="padding:8px; text-align:left; vertical-align:middle; border-bottom-width:1px; border-bottom-style:solid; border-bottom-color:rgb(221,221,221); border-right-width:1px; border-right-style:solid; border-right-color:rgb(238,238,238)"> ~1000ms</td> 
  </tr> 
 </tbody> 
</table>

当我们发现集群中有大key时，要删除时，如何优雅地删除大Key？

## 如何优雅地删除各类大Key ##

从Redis2.8版本开始支持[SCAN][]命令，通过m次时间复杂度为O(1)的方式，遍历包含n个元素的大key.  
这样避免单个O(n)的大命令，导致Redis阻塞。 这里删除大key操作的思想也是如此。

### Delete Large Hash Key ###

通过[hscan命令][hscan]，每次获取500个字段，再用[hdel命令][hdel]，每次删除1个字段。  
Python代码：

<table style="border-collapse:collapse; border-spacing:0px; margin:0px; border:none; font-size:14px; table-layout:fixed"> 
 <tbody> 
  <tr style="background-color:rgb(249,249,249)"> 
   <td style="padding:0px; vertical-align:middle; border:none"> 
    <div>
      1 
    </div> 
    <div>
      2 
    </div> 
    <div>
      3 
    </div> 
    <div>
      4 
    </div> 
    <div>
      5 
    </div> 
    <div>
      6 
    </div> 
    <div>
      7 
    </div> 
    <div>
      8 
    </div> </td> 
   <td style="padding:0px; vertical-align:middle; border:none"> 
    <div> 
     <span style="color:#4271ae"><span style="color:#8959a8">def</span> <span style="color:#3e999f">del_large_hash</span><span style="color:#f5871f">()</span>:</span> 
    </div> 
    <div>
      r = redis.StrictRedis(host= 
     <span style="color:#718c0">'redis-host1'</span>, port= 
     <span style="color:#718c0">6379</span>) 
    </div> 
    <div>
      large_hash_key = 
     <span style="color:#718c0">"xxx"</span> 
     <span style="color:#8e908c">#要删除的大hash键名</span> 
    </div> 
    <div>
      cursor = 
     <span style="color:#718c0">'0'</span> 
    </div> 
    <div> 
     <span style="color:#8959a8">while</span> cursor != 
     <span style="color:#718c0">0</span>: 
    </div> 
    <div>
      cursor, data = r.hscan(large_hash_key, cursor=cursor, count= 
     <span style="color:#718c0">500</span>) 
    </div> 
    <div> 
     <span style="color:#8959a8">for</span> item 
     <span style="color:#8959a8"> in</span> data.items(): 
    </div> 
    <div>
      r.hdel(large_hash_key, item[ 
     <span style="color:#718c0">0</span>]) 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

### Delete Large Set Key ###

删除大set键，使用[sscan命令][sscan]，每次扫描集合中500个元素，再用[srem命令][srem]每次删除一个键  
Python代码：

<table style="border-collapse:collapse; border-spacing:0px; margin:0px; border:none; font-size:14px; table-layout:fixed"> 
 <tbody> 
  <tr style="background-color:rgb(249,249,249)"> 
   <td style="padding:0px; vertical-align:middle; border:none"> 
    <div>
      1 
    </div> 
    <div>
      2 
    </div> 
    <div>
      3 
    </div> 
    <div>
      4 
    </div> 
    <div>
      5 
    </div> 
    <div>
      6 
    </div> 
    <div>
      7 
    </div> 
    <div>
      8 
    </div> </td> 
   <td style="padding:0px; vertical-align:middle; border:none"> 
    <div> 
     <span style="color:#4271ae"><span style="color:#8959a8">def</span> <span style="color:#3e999f">del_large_set</span><span style="color:#f5871f">()</span>:</span> 
    </div> 
    <div>
      r = redis.StrictRedis(host= 
     <span style="color:#718c0">'redis-host1'</span>, port= 
     <span style="color:#718c0">6379</span>) 
    </div> 
    <div>
      large_set_key = 
     <span style="color:#718c0">'xxx'</span> 
     <span style="color:#8e908c"># 要删除的大set的键名</span> 
    </div> 
    <div>
      cursor = 
     <span style="color:#718c0">'0'</span> 
    </div> 
    <div> 
     <span style="color:#8959a8">while</span> cursor != 
     <span style="color:#718c0">0</span>: 
    </div> 
    <div>
      cursor, data = r.sscan(large_set_key, cursor=cursor, count= 
     <span style="color:#718c0">500</span>) 
    </div> 
    <div> 
     <span style="color:#8959a8">for</span> item 
     <span style="color:#8959a8"> in</span> data: 
    </div> 
    <div>
      r.srem(large_size_key, item) 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

### Delete Large List Key ###

删除大的List键，未使用scan命令； 通过[ltrim命令][ltrim]每次删除少量元素。  
Python代码：

<table style="border-collapse:collapse; border-spacing:0px; margin:0px; border:none; font-size:14px; table-layout:fixed"> 
 <tbody> 
  <tr style="background-color:rgb(249,249,249)"> 
   <td style="padding:0px; vertical-align:middle; border:none"> 
    <div>
      1 
    </div> 
    <div>
      2 
    </div> 
    <div>
      3 
    </div> 
    <div>
      4 
    </div> 
    <div>
      5 
    </div> </td> 
   <td style="padding:0px; vertical-align:middle; border:none"> 
    <div> 
     <span style="color:#4271ae"><span style="color:#8959a8">def</span> <span style="color:#3e999f">del_large_list</span><span style="color:#f5871f">()</span>:</span> 
    </div> 
    <div>
      r = redis.StrictRedis(host= 
     <span style="color:#718c0">'redis-host1'</span>, port= 
     <span style="color:#718c0">6379</span>) 
    </div> 
    <div>
      large_list_key = 
     <span style="color:#718c0">'xxx'</span> 
     <span style="color:#8e908c">#要删除的大list的键名</span> 
    </div> 
    <div> 
     <span style="color:#8959a8">while</span> r.llen(large_list_key)&gt; 
     <span style="color:#718c0">0</span>: 
    </div> 
    <div>
      r.ltrim(large_list_key, 
     <span style="color:#718c0"> 0</span>, 
     <span style="color:#718c0">-101</span>) 
     <span style="color:#8e908c"> #每次只删除最右100个元素</span> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

### Delete Large Sorted set key ###

删除大的有序集合键，和List类似，使用sortedset自带的[zremrangebyrank命令][zremrangebyrank],每次删除top 100个元素。  
Python代码：

<table style="border-collapse:collapse; border-spacing:0px; margin:0px; border:none; font-size:14px; table-layout:fixed"> 
 <tbody> 
  <tr style="background-color:rgb(249,249,249)"> 
   <td style="padding:0px; vertical-align:middle; border:none"> 
    <div>
      1 
    </div> 
    <div>
      2 
    </div> 
    <div>
      3 
    </div> 
    <div>
      4 
    </div> 
    <div>
      5 
    </div> </td> 
   <td style="padding:0px; vertical-align:middle; border:none"> 
    <div> 
     <span style="color:#4271ae"><span style="color:#8959a8">def</span> <span style="color:#3e999f">del_large_sortedset</span><span style="color:#f5871f">()</span>:</span> 
    </div> 
    <div>
      r = redis.StrictRedis(host= 
     <span style="color:#718c0">'large_sortedset_key'</span>, port= 
     <span style="color:#718c0">6379</span>) 
    </div> 
    <div>
      large_sortedset_key= 
     <span style="color:#718c0">'xxx'</span> 
    </div> 
    <div> 
     <span style="color:#8959a8">while</span> r.zcard(large_sortedset_key)&gt; 
     <span style="color:#718c0">0</span>: 
    </div> 
    <div>
      r.zremrangebyrank(large_sortedset_key, 
     <span style="color:#718c0">0</span>, 
     <span style="color:#718c0">99</span>) 
     <span style="color:#8e908c">#时间复杂度更低 , 每次删除O(log(N)+100)</span> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

## Redis Lazy Free ##

应该从3.4版本开始，Redis会支持lazy delete free的方式，删除大键的过程不会阻塞正常请求。

参考:  
[lazy redis][]  
[issue 1748][]  
[Delete Large Hash][]

[redis-cluster-imbalance]: https://zhuoroger.github.io/2016/08/03/redis-cluster-imbalance/
[DEL]: http://redis.io/commands/del
[SCAN]: http://redis.io/commands/scan
[hscan]: http://redis.io/commands/hscan
[hdel]: http://redis.io/commands/hdel
[sscan]: http://redis.io/commands/sscan
[srem]: http://redis.io/commands/srem
[ltrim]: http://redis.io/commands/ltrim
[zremrangebyrank]: http://redis.io/commands/zremrangebyrank
[lazy redis]: http://antirez.com/news/93
[issue 1748]: https://github.com/antirez/redis/issues/1748
[Delete Large Hash]: https://www.redisgreen.net/blog/deleting-large-hashes/