python文本向量化对比词的相似度

电玩女神 2023-02-15 10:27 134阅读 0赞

## 文本向量化 ##

##### 创建一个目录，并创建`data`文件夹 #####

### 安装依赖 ###

pip install gensim

### 下载数据集 ###

数据集大概1.2G，下载完成后放在`data`文件夹下

### 数据预处理 ###

#### 创建数据预处理文件 ####

`data_pre_process.py`

# -*- coding: utf-8 -*-
    from gensim.corpora import WikiCorpus
    import jieba
    from langconv import *
    
    def my_function():
        space = ' '
        i = 0
        l = []
        zhwiki_name = './data/zhwiki-latest-pages-articles.xml.bz2'
        f = open('./data/reduce_zhiwiki.txt', 'w')
        wiki = WikiCorpus(zhwiki_name, lemmatize=False, dictionary={ })
        for text in wiki.get_texts():
            for temp_sentence in text:
                temp_sentence = Converter('zh-hans').convert(temp_sentence)
                seg_list = list(jieba.cut(temp_sentence))
                for temp_term in seg_list:
                    l.append(temp_term)
            f.write(space.join(l) + '\n')
            l = []
            i = i + 1
    
            if (i %200 == 0):
                print('Saved ' + str(i) + ' articles')
        f.close()
    
    if __name__ == '__main__':
        my_function()

#### 执行数据预处理程序 ####

python3 data_pre_process.py

由于语料库比较大，过程可能较长，需要耐心等待。

`data`下会产生`reduce_zhiwili.txt`文档

### 训练 ###

#### 创建模型训练文件 ####

`training.py`

# -*- coding: utf-8 -*-
    from gensim.models import Word2Vec
    from gensim.models.word2vec import LineSentence
    import logging
    
    logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
    
    def my_function():
        wiki_news = open('./data/reduce_zhiwiki.txt', 'r')
        model = Word2Vec(LineSentence(wiki_news), sg=0,size=192, window=5, min_count=5, workers=9)
        model.save('zhiwiki_news.word2vec')
    
    if __name__ == '__main__':
        my_function()

#### 开始训练 ####

python3 training.py

训练过程可能较长，需要耐心等待。

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyOTI1MDMx_size_16_color_FFFFFF_t_70_pic_center]

将产生的以`zhiwiki_`开头的文件放在`data`文件夹下

### 测试 ###

#### 创建测试文件 ####

`test.py`

#coding=utf-8
    import gensim
    def my_function():
    
        model = gensim.models.Word2Vec.load('./data/zhiwiki_news.word2vec')
        print(model.similarity('西红柿','番茄'))  #相似度为0.63
        print(model.similarity('西红柿','香蕉'))  #相似度为0.44
    
        word = '中国'
        if word in model.wv.index2word:
            print(model.most_similar(word))
    
    if __name__ == '__main__':
        my_function()

#### 执行测试 ####

python3 test.py

可以看到结果：

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyOTI1MDMx_size_16_color_FFFFFF_t_70_pic_center 1]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyOTI1MDMx_size_16_color_FFFFFF_t_70_pic_center]: https://img-blog.csdnimg.cn/20200605105606879.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyOTI1MDMx,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyOTI1MDMx_size_16_color_FFFFFF_t_70_pic_center 1]: https://img-blog.csdnimg.cn/20200605105548730.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyOTI1MDMx,size_16,color_FFFFFF,t_70#pic_center