NLP系列 1. IMDB和THUCNews数据集数据集的探索

客官°小女子只卖身不卖艺 2022-02-03 19:06 643阅读 0赞

### 文章目录 ###

*  探索IMDB数据集和THUCNews数据集
 *   *  IMDB的探索
     *  THUCNews数据集的探索
     *   *  数据集来源
         *  数据集介绍

# 探索IMDB数据集和THUCNews数据集 #

## IMDB的探索 ##

由keras直接加载数据集,再将数据集中已经预处理过的代表词的数字转换回字词  
代码见 [https://github.com/sherpahu/NLP\_practice/blob/master/Task1/imdb.ipynb][https_github.com_sherpahu_NLP_practice_blob_master_Task1_imdb.ipynb]

主要参考：[https://tensorflow.google.cn/tutorials/keras/basic\_text\_classification][https_tensorflow.google.cn_tutorials_keras_basic_text_classification]

import keras
    imdb=keras.datasets.imdb

Using TensorFlow backend.

(train_data,train_labels),(test_data,test_labels)=imdb.load_data(num_words=10000)
    #num_words=10000表示只保留训练数据集中的最常见的10000个单词，舍弃低频词。防止向量的数据过大，并且保留最有用的信息

train_data[0]

[1,
     14,
     22,
     16,
     43,
     ...
     16,
     5345,
     19,
     178,
     32]

imdb里面是已经处理好的数据，每一个数字代表一个单词，

train_labels[0]

len(train_data),len(train_labels),len(test_data),len(test_labels)

(25000, 25000, 25000, 25000)

import pandas as pd
    series=pd.Series(train_labels)
    series.value_counts()

1    12500
    0    12500
    dtype: int64

0代表负面评价，1代表正面  
由上面的train\_labels的统计可以看出，正面、负面各占一半下面尝试将数字代表的评价转化为原文

# A dictionary mapping words to an integer index
    word_index=imdb.get_word_index()
    
    # The first indices are reserved
    word_index = { k:(v+3) for k,v in word_index.items()}
    word_index["<PAD>"] = 0
    word_index["<START>"] = 1
    word_index["<UNK>"] = 2  # unknown
    word_index["<UNUSED>"] = 3
    
    index2word=dict([(value,key) for (key,value) in word_index.items()])
    def decode_review(text):
        return ' '.join([index2word.get(i,'?') for i in text])

decode_review(train_data[0])

"<START> this film was just brilliant casting location scenery story direction everyone's really suited the part they played and you could just imagine being there robert <UNK> is an amazing actor and now the same being director <UNK> father came from the same scottish island as myself so i loved the fact there was a real connection with this film the witty remarks throughout the film were great it was just brilliant so much that i bought the film as soon as it was released for <UNK> and would recommend it to everyone to watch and the fly fishing was amazing really cried at the end it was so sad and you know what they say if you cry at a film it must have been good and this definitely was also <UNK> to the two little boy's that played the <UNK> of norman and paul they were just brilliant children are often left out of the <UNK> list i think because the stars that play them all grown up are such a big profile for the whole film but these children are amazing and should be praised for what they have done don't you think the whole story was so lovely because it was true and was someone's life after all that was shared with us all"

## THUCNews数据集的探索 ##

### 数据集来源 ###

完整数据集： [http://thuctc.thunlp.org/\#中文文本分类数据集THUCNews][http_thuctc.thunlp.org_THUCNews]

子数据集： 链接: [https://pan.baidu.com/s/1hugrfRu][https_pan.baidu.com_s_1hugrfRu] 密码: qfud

子数据集来源：[https://github.com/gaussic/text-classification-cnn-rnn][https_github.com_gaussic_text-classification-cnn-rnn]

> 数据集划分如下：
> 
>  *  训练集: 5000\*10
>  *  验证集: 500\*10
>  *  测试集: 1000\*10
> 
> 从原数据集生成子集的过程请参看`helper`下的两个脚本。其中，`copy_data.sh`用于从每个分类拷贝6500个文件，`cnews_group.py`用于将多个文件整合到一个文件中。执行该文件后，得到三个数据文件：
> 
>  *  cnews.train.txt: 训练集(50000条)
>  *  cnews.val.txt: 验证集(5000条)
>  *  cnews.test.txt: 测试集(10000条)

### 数据集介绍 ###

THUCNews是根据新浪新闻RSS订阅频道2005~2011年间的历史数据筛选过滤生成，包含74万篇新闻文档（2.19 GB），均为UTF-8纯文本格式。我们在原始新浪新闻分类体系的基础上，重新整合划分出14个候选分类类别：财经、彩票、房产、股票、家居、教育、科技、社会、时尚、时政、体育、星座、游戏、娱乐。使用THUCTC工具包在此数据集上进行评测，准确率可以达到88.6%。

[https_github.com_sherpahu_NLP_practice_blob_master_Task1_imdb.ipynb]: https://github.com/sherpahu/NLP_practice/blob/master/Task1/imdb.ipynb
[https_tensorflow.google.cn_tutorials_keras_basic_text_classification]: https://tensorflow.google.cn/tutorials/keras/basic_text_classification
[http_thuctc.thunlp.org_THUCNews]: http://thuctc.thunlp.org/#%E4%B8%AD%E6%96%87%E6%96%87%E6%9C%AC%E5%88%86%E7%B1%BB%E6%95%B0%E6%8D%AE%E9%9B%86THUCNews
[https_pan.baidu.com_s_1hugrfRu]: https://pan.baidu.com/s/1hugrfRu
[https_github.com_gaussic_text-classification-cnn-rnn]: https://github.com/gaussic/text-classification-cnn-rnn