java操作数据入ES

ゞ浴缸里的玫瑰 2023-06-08 06:59 22阅读 0赞

//todo

es更多的功能在于检索统计上,其查询和统计的速度相对于结构性数据库更强,更便捷,但是其非结构化的数据格式也是入库的一个难题,此外,es在对于save和update操作时比较耗性能的...但是,这个库肯定是要入的,所以抛开数据库本身的性能,我们要在代码除了此精简结构,使入库最优化...

此时ES入库操作的类也是JestClient,自动注入;

定义个实体类方便接收转化数据

public class EsObj implements Serializable {
    
    
        private static final long serialVersionUID = 1994386320080028446L;
        private HashMap fieldMap;
        private String index;
        private String indexType;
        private String indexId;
    	}

创建es对象的格式

public EsObj createEsObj(String[] msgArr, StorageParam storageParam) {
            EsObj esObj = new EsObj();
            HashMap<String, String> fieldMap = new HashMap<String, String>();
            for (int i = 0; i < msgArr.length; i++) {
                fieldMap.put(storageParam.getMsgFieldsArr()[i], msgArr[i]);
            }
            esObj.setFieldMap(fieldMap);
            //默认类型和索引一样
            esObj.setIndex(storageParam.getEsIndex());
            esObj.setIndexType(storageParam.getEsIndex());
            if (storageParam.isUniqueIndex()) {
                StringBuffer indexIdBuf = new StringBuffer();
                String[] indexIdArr = storageParam.getEsIndexId().split(",", -1);
                for (String field : indexIdArr) {
                    indexIdBuf.append(fieldMap.get(field));
                }
                esObj.setIndexId(indexIdBuf.toString());
            } else {
                esObj.setIndexId("");
            }
            return esObj;
        }

数据操作处理类

@Service
    public class EsRepository {
    
        private static final Logger log = LoggerFactory.getLogger(EsRepository.class);
    
        @Autowired
        private JestClient jestClient;
    
        /**
         * 单个内容保存到es
         *
         * @param esObj es对象
         */
        public void save(EsObj esObj, Boolean isUniqueIndex) {
            Index index;
            if (isUniqueIndex) {
                index = new Index.Builder(esObj.getFieldMap()).index(esObj.getIndex()).type(esObj.getIndexType()).id(esObj.getIndexId()).build();
            } else {
                index = new Index.Builder(esObj.getFieldMap()).index(esObj.getIndex()).type(esObj.getIndexType()).build();
            }
            try {
                jestClient.execute(index);
                log.debug("ES 插入成功");
            } catch (IOException e) {
                log.error("Exception trace:",e.getMessage(), e);
            }
        }
    
    
        /**
         * 批量保存内容到ES 增加错误重试功能
         *
         * @param esObjList 数据列表
         * @param isUniqueIndex 是否有唯一索引
         * @param retryTimes 重试次数
         */
        public void batchSave(List<EsObj> esObjList, Boolean isUniqueIndex,int retryTimes) {
            boolean isException = false;
            Bulk.Builder bulk = new Bulk.Builder();
            Index index;
            if (isUniqueIndex) {
                for (EsObj esObj : esObjList) {
                    index = new Index.Builder(esObj.getFieldMap()).index(esObj.getIndex()).type(esObj.getIndexType()).id(esObj.getIndexId()).build();
                    bulk.addAction(index);
                }
            } else {
                for (EsObj esObj : esObjList) {
                    index = new Index.Builder(esObj.getFieldMap()).index(esObj.getIndex()).type(esObj.getIndexType()).build();
                    bulk.addAction(index);
                }
            }
            try {
                jestClient.execute(bulk.build());
                log.debug("ES 插入条数:"+esObjList.size());
            } catch (IOException e) {
                log.error("ES batchSave:"+e.getMessage()+"retryTimes:"+retryTimes);
                isException  = true;
            }
            //错误重试
            if(isException && retryTimes > 0){
                batchSave( esObjList, isUniqueIndex,--retryTimes);
            }
        }
    }

简单的基础的操作就是这样,具体的还需自己去潜在优化,至于自己封装的对象的大同小异处还是因人而异的...自己只是个弟弟,还望大佬指正......