NLP（二十三）序列标注算法评估模块seqeval的使用

妖狐艹你老母 2023-07-13 05:54 45阅读 0赞

在NLP中，序列标注算法是常见的深度学习模型，但是，对于序列标注算法的评估，我们真的熟悉吗？  
  在本文中，笔者将会序列标注算法的模型效果评估方法和`seqeval`的使用。

### 序列标注算法的模型效果评估 ###

在序列标注算法中，一般我们会形成如下的序列列表，如下：

['O', 'O', 'B-MISC', 'I-MISC', 'B-MISC', 'I-MISC', 'O', 'B-PER', 'I-PER']

一般序列标注算法的格式有`BIO`，`IOBES`，`BMES`等。其中，`实体`指的是从B开头标签开始的，同一类型（比如：PER/LOC/ORG）的，非O的连续标签序列。  
  常见的序列标注算法的模型效果评估指标有准确率（accuracy）、查准率(percision)、召回率(recall)、F1值等，计算的公式如下：

*  准确率: accuracy = 预测对的元素个数/总的元素个数
 *  查准率：precision = 预测正确的实体个数 / 预测的实体总个数
 *  召回率：recall = 预测正确的实体个数 / 标注的实体总个数
 *  F1值：F1 = 2 \*准确率 \* 召回率 / (准确率 + 召回率)

举个例子，我们有如下的真实序列`y_true`和预测序列`y_pred`，如下：

y_true = ['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O', 'B-PER', 'I-PER']
    y_pred = ['O', 'O', 'B-MISC', 'I-MISC', 'B-MISC', 'I-MISC', 'O', 'B-PER', 'I-PER']

列表中一个有9个元素，其中预测对的元素个数为6个，那么准确率为2/3。标注的实体总个数为2个，预测的实体总个数为3个，预测正确的实体个数为1个，那么precision=1/3, recall=1/2, F1=0.4。

### seqeval的使用 ###

一般我们的序列标注算法，是用`conlleval.pl`脚本实现，但这是用perl语言实现的。在Python中，也有相应的序列标注算法的模型效果评估的第三方模块，那就是`seqeval`，其官网网址为：[https://pypi.org/project/seqeval/0.0.3/][https_pypi.org_project_seqeval_0.0.3] 。  
  `seqeval`支持`BIO`，`IOBES`标注模式，可用于命名实体识别，词性标注，语义角色标注等任务的评估。  
  官网文档中给出了两个例子，笔者修改如下：  
  例子1：

# -*- coding: utf-8 -*-
    from seqeval.metrics import f1_score
    from seqeval.metrics import precision_score
    from seqeval.metrics import accuracy_score
    from seqeval.metrics import recall_score
    from seqeval.metrics import classification_report
    
    y_true = ['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O', 'B-PER', 'I-PER']
    y_pred = ['O', 'O', 'B-MISC', 'I-MISC', 'B-MISC', 'I-MISC', 'O', 'B-PER', 'I-PER']
    
    print("accuary: ", accuracy_score(y_true, y_pred))
    print("p: ", precision_score(y_true, y_pred))
    print("r: ", recall_score(y_true, y_pred))
    print("f1: ", f1_score(y_true, y_pred))
    print("classification report: ")
    print(classification_report(y_true, y_pred))

输出结果如下：

accuary:  0.6666666666666666
    p:  0.3333333333333333
    r:  0.5
    f1:  0.4
    classification report: 
               precision    recall  f1-score   support
    
         MISC       0.00      0.00      0.00         1
          PER       1.00      1.00      1.00         1
    
    micro avg       0.33      0.50      0.40         2
    macro avg       0.50      0.50      0.50         2

例子2：

# -*- coding: utf-8 -*-
    from seqeval.metrics import f1_score
    from seqeval.metrics import precision_score
    from seqeval.metrics import accuracy_score
    from seqeval.metrics import recall_score
    from seqeval.metrics import classification_report
    
    y_true = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER']]
    y_pred =  [['O', 'O', 'B-MISC', 'I-MISC', 'B-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER']]
    
    print("accuary: ", accuracy_score(y_true, y_pred))
    print("p: ", precision_score(y_true, y_pred))
    print("r: ", recall_score(y_true, y_pred))
    print("f1: ", f1_score(y_true, y_pred))
    print("classification report: ")
    print(classification_report(y_true, y_pred))

输出结果同上。

### 在Keras中使用seqeval ###

笔者一年多年写过文章：[用深度学习实现命名实体识别（NER）][NER]， 我们对模型训练部分的代码加以改造，使之在训练过程中能输出F1值。  
  在Github上下载项目`DL_4_NER`，网址为：[https://github.com/percent4/DL\_4\_NER][https_github.com_percent4_DL_4_NER] 。修改utils.py中的文件夹路径，以及模型训练部分的代码（DL\_4\_NER/Bi\_LSTM\_Model\_training.py）如下：

# -*- coding: utf-8 -*-
    import pickle
    import numpy as np
    import pandas as pd
    from utils import BASE_DIR, CONSTANTS, load_data
    from data_processing import data_processing
    from keras.utils import np_utils, plot_model
    from keras.models import Sequential
    from keras.preprocessing.sequence import pad_sequences
    from keras.layers import Bidirectional, LSTM, Dense, Embedding, TimeDistributed
    
    
    # 模型输入数据
    def input_data_for_model(input_shape):
    
        # 数据导入
        input_data = load_data()
        # 数据处理
        data_processing()
        # 导入字典
        with open(CONSTANTS[1], 'rb') as f:
            word_dictionary = pickle.load(f)
        with open(CONSTANTS[2], 'rb') as f:
            inverse_word_dictionary = pickle.load(f)
        with open(CONSTANTS[3], 'rb') as f:
            label_dictionary = pickle.load(f)
        with open(CONSTANTS[4], 'rb') as f:
            output_dictionary = pickle.load(f)
        vocab_size = len(word_dictionary.keys())
        label_size = len(label_dictionary.keys())
    
        # 处理输入数据
        aggregate_function = lambda input: [(word, pos, label) for word, pos, label in
                                                zip(input['word'].values.tolist(),
                                                    input['pos'].values.tolist(),
                                                    input['tag'].values.tolist())]
    
        grouped_input_data = input_data.groupby('sent_no').apply(aggregate_function)
        sentences = [sentence for sentence in grouped_input_data]
    
        x = [[word_dictionary[word[0]] for word in sent] for sent in sentences]
        x = pad_sequences(maxlen=input_shape, sequences=x, padding='post', value=0)
        y = [[label_dictionary[word[2]] for word in sent] for sent in sentences]
        y = pad_sequences(maxlen=input_shape, sequences=y, padding='post', value=0)
        y = [np_utils.to_categorical(label, num_classes=label_size + 1) for label in y]
    
        return x, y, output_dictionary, vocab_size, label_size, inverse_word_dictionary
    
    
    # 定义深度学习模型：Bi-LSTM
    def create_Bi_LSTM(vocab_size, label_size, input_shape, output_dim, n_units, out_act, activation):
        model = Sequential()
        model.add(Embedding(input_dim=vocab_size + 1, output_dim=output_dim,
                            input_length=input_shape, mask_zero=True))
        model.add(Bidirectional(LSTM(units=n_units, activation=activation,
                                     return_sequences=True)))
        model.add(TimeDistributed(Dense(label_size + 1, activation=out_act)))
        model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
        return model
    
    
    # 模型训练
    def model_train():
    
        # 将数据集分为训练集和测试集，占比为9:1
        input_shape = 60
        x, y, output_dictionary, vocab_size, label_size, inverse_word_dictionary = input_data_for_model(input_shape)
        train_end = int(len(x)*0.9)
        train_x, train_y = x[0:train_end], np.array(y[0:train_end])
        test_x, test_y = x[train_end:], np.array(y[train_end:])
    
        # 模型输入参数
        activation = 'selu'
        out_act = 'softmax'
        n_units = 100
        batch_size = 32
        epochs = 10
        output_dim = 20
    
        # 模型训练
        lstm_model = create_Bi_LSTM(vocab_size, label_size, input_shape, output_dim, n_units, out_act, activation)
        lstm_model.fit(train_x, train_y, validation_data=(test_x, test_y), epochs=epochs, batch_size=batch_size, verbose=1)
    
    
    model_train()

模型训练的结果如下（中间过程省略）：

......
    12598/12598 [==============================] - 26s 2ms/step - loss: 0.0075 - acc: 0.9981 - val_loss: 0.2131 - val_acc: 0.9592

我们修改代码，在lstm\_model.fit那一行修改代码如下：

lables = ['O', 'B-MISC', 'I-MISC', 'B-ORG', 'I-ORG', 'B-PER', 'B-LOC', 'I-PER', 'I-LOC', 'sO']
        id2label = dict(zip(range(len(lables)), lables))
        callbacks = [F1Metrics(id2label)]
        lstm_model.fit(train_x, train_y, validation_data=(test_x, test_y), epochs=epochs,
                       batch_size=batch_size, verbose=1, callbacks=callbacks)

此时输出结果为：

12598/12598 [==============================] - 26s 2ms/step - loss: 0.0089 - acc: 0.9978 - val_loss: 0.2145 - val_acc: 0.9560
     - f1: 95.40
               precision    recall  f1-score   support
    
         MISC     0.9707    0.9833    0.9769     15844
          PER     0.9080    0.8194    0.8614      1157
          LOC     0.7517    0.8095    0.7795       677
          ORG     0.8290    0.7289    0.7757       745
           sO     0.7757    0.8300    0.8019       100
    
    micro avg     0.9524    0.9556    0.9540     18523
    macro avg     0.9520    0.9556    0.9535     18523

这就是seqeval的强大之处。  
  关于seqeval在Keras的使用，有不清楚的地方可以参考该项目的Github网址：[https://github.com/chakki-works/seqeval][https_github.com_chakki-works_seqeval] 。

### 总结 ###

感谢大家的阅读，本次分享到此结束。  
  欢迎大家关注我的微信公众号：`Python爬虫与算法`。

### 参考网址 ###

1.  序列标注的准确率和召回率计算: [https://zhuanlan.zhihu.com/p/56582082][https_zhuanlan.zhihu.com_p_56582082]
2.  seqeval官方文档： [https://pypi.org/project/seqeval/0.0.3/][https_pypi.org_project_seqeval_0.0.3]

[https_pypi.org_project_seqeval_0.0.3]: https://pypi.org/project/seqeval/0.0.3/
[NER]: https://blog.csdn.net/jclian91/article/details/84142304
[https_github.com_percent4_DL_4_NER]: https://github.com/percent4/DL_4_NER
[https_github.com_chakki-works_seqeval]: https://github.com/chakki-works/seqeval
[https_zhuanlan.zhihu.com_p_56582082]: https://zhuanlan.zhihu.com/p/56582082