Redis key模糊查询聚合实现多维度查询数据可视化

电玩女神 2022-01-30 12:27 640阅读 0赞

一个分布式多租户多维度需要统计聚合数据的可视化，通过关系型数据库查询效率实在太低尤其是年、季度、月统计数据，采用redis的key模糊查询可以实现对一类数据的多维度筛选和统计。可以做到年、季度、月、日、租户编码、业务类型、业务ID、业务状态、操作人等等维度的聚合查询。

**目录**

Key模糊聚合查询

测试代码和初始化改造

政区初始化及统计流程设计

--------------------

本文基于：[Redis连接池及单节点工具与多Redis节点管理工具 ][Redis_Redis_]

## Key模糊聚合查询 ##

**Jedis keys方法：**

//keys
        public static Set<String> keys(String key){
        	Jedis jedis = null;
        	Set<String> result = null;
        	try {
        		jedis = RedisPool.getJedis();
        		result = jedis.keys(key);
        	} catch (Exception e) {
        		e.printStackTrace();
        	} finally {
        		if (jedis != null) jedis.close();
        	}
        	return result;
        }

**通过上一次对Redis的多节点的封装，再进行补充说明，添加节点追加和移除功能：**

package com.boonya.webservice.util;
    
    import java.util.concurrent.ConcurrentHashMap;
    import redis.clients.jedis.Jedis;
    import redis.clients.jedis.JedisPool;
    import com.boonya.common.StringUtils;
    import com.boonya.util.Util;
    /**
     * 
     * @author PJL
     *
     * @note     功能描述:TODO Redis节点管理
     * @package  com.boonya.webservice.util
     * @filename RedisNodeManger.java
     * @date     2019年5月9日 下午6:57:04
     */
    public class RedisNodeManger {
    	
    	private static boolean  dwRedisFilter =false ;
    	
    	private static ConcurrentHashMap<String, JedisPool> redisNodes=new ConcurrentHashMap<String, JedisPool>();
    	
    	static {
    		init();
    	}
    	
    	public static boolean isDwRedisFilter(){
    		return dwRedisFilter;
    	}
    	
    	private static void init(){
    		String connections=Util.getConfig("dwRedisConnections");
    		if(!StringUtils.IsNullOrEmpty(connections)){
    			String nodes[]=connections.split(",");
    			for (String node : nodes) {
    				parseNode(node);
    			}
    			if(redisNodes.size()>0){
    				dwRedisFilter=true;
    			}
    		}
    	}
    	
    	public static void appendNode(String connections){
    		if(!StringUtils.IsNullOrEmpty(connections)){
    			String nodes[]=connections.split(",");
    			for (String node : nodes) {
    				parseNode(node);
    			}
    			if(redisNodes.size()>0){
    				dwRedisFilter=true;
    			}
    		}
    	}
    	
    	public static void removeNode(String dwCode){
    		if(redisNodes.containsKey(dwCode)){
    			redisNodes.remove(dwCode);
    		}
    	}
    	
    	private static void parseNode(String node){
    		String[] values=node.split("#");
    		String dwCode=values[0];
    		String host=values[1];
    		String port=values[2];
    		String auth=values[3];
    		JedisPool pool=RedisPool.generateJedisPool(host, port, auth);
    		if(!StringUtils.IsNullOrEmpty(dwCode)){
    			if(pool!=null)redisNodes.put(dwCode, pool);
    		}
    	}
    	
    	public static Jedis getJedis(String dwCode){
    		if(redisNodes.containsKey(dwCode)){
    			return redisNodes.get(dwCode).getResource();
    		}
    		return null;
    	}
    	
    	public static boolean hasRedisNode(String dwCode){
    		return redisNodes.containsKey(dwCode);
    	}
    
    }

**编写测试类观察数据聚合查询情况：**

package test;
    
    
    import java.util.Set;
    import com.boonya.webservice.util.RedisNodeManagerUtil;
    import com.boonya.webservice.util.RedisNodeManger;
    
    public class RedisTest {
    	
    	public static void main(String[] args) {
    		String node="test";
    		RedisNodeManger.appendNode("test#192.168.20.10#6380#dlwy@2019");
    		
    		Set<String> keys = RedisNodeManagerUtil.keys(node,"beijing:haidian:2019:5:19:12:01:10000");
    		
    		System.out.println("beijing:haidian:2019:5:19:12:01:10000 "+keys.size());
    		
            keys = RedisNodeManagerUtil.keys(node,"beijing:haidian:2019:5:19:12:01:*");
    		
    		System.out.println("beijing:haidian:2019:5:19:12:01:* "+keys.size());
    		
    		
    		keys = RedisNodeManagerUtil.keys(node,"beijing:haidian:2019:5:19:12:*:*");
    			
    		System.out.println("beijing:haidian:2019:5:19:12:*:* "+keys.size());
    		
    		keys = RedisNodeManagerUtil.keys(node,"beijing:haidian:2019:*:19:12:*:*");
    		
    		System.out.println("beijing:haidian:2019:*:19:12:*:* "+keys.size());
    		
    		
    		keys = RedisNodeManagerUtil.keys(node,"beijing:xicheng:*:*:*:*:*:*");
    		
    		System.out.println("beijing:xicheng:*:*:*:*:*:* "+keys.size());
    		
    		
    	    keys = RedisNodeManagerUtil.keys(node,"beijing:*:*:*:*:*:*:*");
    		
    		System.out.println("beijing:*:*:*:*:*:*:* "+keys.size());
    	}
    
    }

Redis数据：

RDM Redis Console
    连接中...
    Connected.
    192.168.20.10：6380:0>keys *
     1)  "beijing:haidian:2019:5:19:12:01:10000"
    192.168.20.10：6380:0>set beijing:xicheng:2019:5:19:13:22:10001 2
    "OK"
    192.168.20.10：6380:0>keys *
     1)  "beijing:xicheng:2019:5:19:13:22:10001"
     2)  "beijing:haidian:2019:5:19:12:01:10000"
    192.168.20.10：6380:0>

运行结果：

Redis未初始化
    192.168.1.111:6379 redis节点初始化完成
    192.168.20.10:6380 redis节点初始化完成
    beijing:haidian:2019:5:19:12:01:10000 1
    beijing:haidian:2019:5:19:12:01:* 1
    beijing:haidian:2019:5:19:12:*:* 1
    beijing:haidian:2019:*:19:12:*:* 1
    beijing:xicheng:*:*:*:*:*:* 1
    beijing:*:*:*:*:*:*:* 2

条件随便写的，上面多维统计聚合结果符合预期。

## 测试代码和初始化改造 ##

**redis 节点管理类：**

package com.boonya.webservice.util;
    
    import java.util.concurrent.ConcurrentHashMap;
    import redis.clients.jedis.Jedis;
    import redis.clients.jedis.JedisPool;
    import com.boonya.common.StringUtils;
    import com.boonya.util.Util;
    /**
     * 
     * @author PJL
     *
     * @note     功能描述:TODO Redis节点管理
     * @package  com.boonya.webservice.util
     * @filename RedisNodeManger.java
     * @date     2019年5月9日 下午6:57:04
     */
    public class RedisNodeManger {
    	
    	private static boolean  init =false ;
    	
    	private static boolean  dwRedisFilter =false ;
    	
    	private static ConcurrentHashMap<String, JedisPool> redisNodes=new ConcurrentHashMap<String, JedisPool>();
    	
    	static {
    		init();
    	}
    	
    	public static boolean isDwRedisFilter(){
    		return dwRedisFilter;
    	}
    	
    	public static void init(){
    		if(!init){
    			String connections=Util.getConfig("dwRedisConnections");
    			if(!StringUtils.IsNullOrEmpty(connections)){
    				String nodes[]=connections.split(",");
    				for (String node : nodes) {
    					parseNode(node);
    				}
    				if(redisNodes.size()>0){
    					dwRedisFilter=true;
    				}
    			}
    			init=true;
    		}
    	}
    	
    	public static void appendNode(String connections){
    		if(!StringUtils.IsNullOrEmpty(connections)){
    			String nodes[]=connections.split(",");
    			for (String node : nodes) {
    				parseNode(node);
    			}
    			if(redisNodes.size()>0){
    				dwRedisFilter=true;
    			}
    		}
    	}
    	
    	public static void removeNode(String dwCode){
    		if(redisNodes.containsKey(dwCode)){
    			redisNodes.remove(dwCode);
    		}
    	}
    	
    	private static void parseNode(String node){
    		String[] values=node.split("#");
    		String dwCode=values[0];
    		String host=values[1];
    		String port=values[2];
    		String auth=values[3];
    		JedisPool pool=RedisPool.generateJedisPool(host, port, auth);
    		if(!StringUtils.IsNullOrEmpty(dwCode)){
    			if(pool!=null)redisNodes.put(dwCode, pool);
    		}
    	}
    	
    	public static Jedis getJedis(String dwCode){
    		if(redisNodes.containsKey(dwCode)){
    			return redisNodes.get(dwCode).getResource();
    		}
    		return null;
    	}
    	
    	public static boolean hasRedisNode(String dwCode){
    		return redisNodes.containsKey(dwCode);
    	}
    
    }

**测试类：**

package test;
    
    
    import java.util.Set;
    import com.boonya.webservice.util.RedisNodeManagerUtil;
    import com.boonya.webservice.util.RedisNodeManger;
    
    public class RedisTest {
    	
    	public static void main(String[] args) {
    		String node="13304319115";
    		
    		RedisNodeManger.init();
    		
    		RedisNodeManagerUtil.set(node,"beijing:haidian:2019:5:19:12:01:10000","haidian_service");
    		RedisNodeManagerUtil.set(node,"beijing:xicheng:2019:5:19:12:02:10001","xicheng_service");
    		
    		Set<String> keys = RedisNodeManagerUtil.keys(node,"beijing:haidian:2019:5:19:12:01:10000");
    		
    		System.out.println("beijing:haidian:2019:5:19:12:01:10000 "+keys.size());
    		
            keys = RedisNodeManagerUtil.keys(node,"beijing:haidian:2019:5:19:12:01:*");
    		
    		System.out.println("beijing:haidian:2019:5:19:12:01:* "+keys.size());
    		
    		
    		keys = RedisNodeManagerUtil.keys(node,"beijing:haidian:2019:5:19:12:*:*");
    			
    		System.out.println("beijing:haidian:2019:5:19:12:*:* "+keys.size());
    		
    		keys = RedisNodeManagerUtil.keys(node,"beijing:haidian:2019:*:19:12:*:*");
    		
    		System.out.println("beijing:haidian:2019:*:19:12:*:* "+keys.size());
    		
    		
    		keys = RedisNodeManagerUtil.keys(node,"beijing:xicheng:*:*:*:*:*:*");
    		
    		System.out.println("beijing:xicheng:*:*:*:*:*:* "+keys.size());
    		
    		
    	    keys = RedisNodeManagerUtil.keys(node,"beijing:*:*:*:*:*:*:*");
    		
    		System.out.println("beijing:*:*:*:*:*:*:* "+keys.size());
    	}
    
    }

注：之所以要改造init就是为了防止redis被多次初始化，也无需在append额外的redis节点来测试了，需要就init。

## 政区初始化及统计流程设计 ##

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Jvb255YQ_size_16_color_FFFFFF_t_70][]

[Redis_Redis_]: https://blog.csdn.net/boonya/article/details/90051969
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Jvb255YQ_size_16_color_FFFFFF_t_70]: /images/20220130/ba77b466550a4c309c90ad3243ae37e3.png