NLP-文本分类：Bert文本分类(fine-tuning)【一分类（MSELoss）、多分类（CrossEntropyLoss）、多标签分类（BCEWithLogitsLoss）】

绝地灬酷狼 2023-10-01 19:52 158阅读 0赞

![在这里插入图片描述][watermark_type_d3F5LXplbmhlaQ_shadow_50_text_Q1NETiBA5b-N6ICF44Gu5Lmx5aSq6YOO_size_20_color_FFFFFF_t_70_g_se_x_16]

## 二、Bert源码（BertForSequenceClassification） ##

源码位置：\\transformers\\models\\bert\\modeling\_bert.py

@add_start_docstrings(
        """
        Bert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled
        output) e.g. for GLUE tasks.
        """,
        BERT_START_DOCSTRING,
    )
    class BertForSequenceClassification(BertPreTrainedModel):
        def __init__(self, config):
            super().__init__(config)
            self.num_labels = config.num_labels
            self.config = config
    
            self.bert = BertModel(config)
            classifier_dropout = (
                config.classifier_dropout if config.classifier_dropout is not None else config.hidden_dropout_prob
            )
            self.dropout = nn.Dropout(classifier_dropout)
            self.classifier = nn.Linear(config.hidden_size, config.num_labels)
    
            # Initialize weights and apply final processing
            self.post_init()
    
        @add_start_docstrings_to_model_forward(BERT_INPUTS_DOCSTRING.format("batch_size, sequence_length"))
        @add_code_sample_docstrings(
            processor_class=_TOKENIZER_FOR_DOC,
            checkpoint=_CHECKPOINT_FOR_DOC,
            output_type=SequenceClassifierOutput,
            config_class=_CONFIG_FOR_DOC,
        )
        def forward(
            self,
            input_ids=None,
            attention_mask=None,
            token_type_ids=None,
            position_ids=None,
            head_mask=None,
            inputs_embeds=None,
            labels=None,
            output_attentions=None,
            output_hidden_states=None,
            return_dict=None,
        ):
            r"""
            labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
                Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
                config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
                `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
            """
            return_dict = return_dict if return_dict is not None else self.config.use_return_dict
    
            outputs = self.bert(
                input_ids,
                attention_mask=attention_mask,
                token_type_ids=token_type_ids,
                position_ids=position_ids,
                head_mask=head_mask,
                inputs_embeds=inputs_embeds,
                output_attentions=output_attentions,
                output_hidden_states=output_hidden_states,
                return_dict=return_dict,
            )
    
            pooled_output = outputs[1]
    
            pooled_output = self.dropout(pooled_output)
            logits = self.classifier(pooled_output)
    
            loss = None
            if labels is not None:
                if self.config.problem_type is None:
                    if self.num_labels == 1:
                        self.config.problem_type = "regression"
                    elif self.num_labels > 1 and (labels.dtype == torch.long or labels.dtype == torch.int):
                        self.config.problem_type = "single_label_classification"
                    else:
                        self.config.problem_type = "multi_label_classification"
    
                if self.config.problem_type == "regression":
                    loss_fct = MSELoss()
                    if self.num_labels == 1:
                        loss = loss_fct(logits.squeeze(), labels.squeeze())
                    else:
                        loss = loss_fct(logits, labels)
                elif self.config.problem_type == "single_label_classification":
                    loss_fct = CrossEntropyLoss()
                    loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
                elif self.config.problem_type == "multi_label_classification":
                    loss_fct = BCEWithLogitsLoss()
                    loss = loss_fct(logits, labels)
            if not return_dict:
                output = (logits,) + outputs[2:]
                return ((loss,) + output) if loss is not None else output
    
            return SequenceClassifierOutput(
                loss=loss,
                logits=logits,
                hidden_states=outputs.hidden_states,
                attentions=outputs.attentions,
            )

We will adapt BertForSequenceClassification class to cater for multi-label classification.

class BertForMultiLabelSequenceClassification(PreTrainedBertModel):
        """BERT model for classification.
        This module is composed of the BERT model with a linear layer on top of
        the pooled output.
        """
        def __init__(self, config, num_labels=2):
            super(BertForMultiLabelSequenceClassification, self).__init__(config)
            self.num_labels = num_labels
            self.bert = BertModel(config)
            self.dropout = torch.nn.Dropout(config.hidden_dropout_prob)
            self.classifier = torch.nn.Linear(config.hidden_size, num_labels)
            self.apply(self.init_bert_weights)
    
        def forward(self, input_ids, token_type_ids=None, attention_mask=None, labels=None):
            _, pooled_output = self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False)
            pooled_output = self.dropout(pooled_output)
            logits = self.classifier(pooled_output)
    
            if labels is not None:
                loss_fct = BCEWithLogitsLoss()
                loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1, self.num_labels))
                return loss
            else:
                return logits
            
        def freeze_bert_encoder(self):
            for param in self.bert.parameters():
                param.requires_grad = False
        
        def unfreeze_bert_encoder(self):
            for param in self.bert.parameters():
                param.requires_grad = True

The primary change here is the usage of Binary cross-entropy with logits (BCEWithLogitsLoss) loss function instead of vanilla cross-entropy loss (CrossEntropyLoss) that is used for multiclass classification. Binary cross-entropy loss allows our model to assign independent probabilities to the labels.

The model summary is shows the layers of the model alongwith their dimensions.

BertForMultiLabelSequenceClassification(
      (bert): BertModel(
        (embeddings): BertEmbeddings(
          (word_embeddings): Embedding(28996, 768)
          (position_embeddings): Embedding(512, 768)
          (token_type_embeddings): Embedding(2, 768)
          (LayerNorm): FusedLayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)
          (dropout): Dropout(p=0.1)
        )
        (encoder): BertEncoder(
          (layer): ModuleList(
    #       12 BertLayers
            (11): BertLayer(
              (attention): BertAttention(
                (self): BertSelfAttention(
                  (query): Linear(in_features=768, out_features=768, bias=True)
                  (key): Linear(in_features=768, out_features=768, bias=True)
                  (value): Linear(in_features=768, out_features=768, bias=True)
                  (dropout): Dropout(p=0.1)
                )
                (output): BertSelfOutput(
                  (dense): Linear(in_features=768, out_features=768, bias=True)
                  (LayerNorm): FusedLayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)
                  (dropout): Dropout(p=0.1)
                )
              )
              (intermediate): BertIntermediate(
                (dense): Linear(in_features=768, out_features=3072, bias=True)
              )
              (output): BertOutput(
                (dense): Linear(in_features=3072, out_features=768, bias=True)
                (LayerNorm): FusedLayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1)
              )
            )
          )
        )
        (pooler): BertPooler(
          (dense): Linear(in_features=768, out_features=768, bias=True)
          (activation): Tanh()
        )
      )
      (dropout): Dropout(p=0.1)
      (classifier): Linear(in_features=768, out_features=6, bias=True)
    )

*  BertEmbeddings: Input embedding layer
 *  BertEncoder: The 12 BERT attention layers
 *  Classifier: Our multi-label classifier with out\_features=6, each corresponding to our 6 labels

## Evaluation Metrics ##

We adapted the accuracy metric function to include a threshold, which is set to 0.5 as default.

在这里插入代码片

--------------------

参考资料：  
[Multi-label Text Classification using BERT – The Mighty Transformer][Multi-label Text Classification using BERT _ The Mighty Transformer]  
[https://github.com/huggingface/transformers][https_github.com_huggingface_transformers]  
[Bert文本分类(fine-tuning)][Bert_fine-tuning]  
[干货 | BERT fine-tune 终极实践教程][_ BERT fine-tune]  
[Bert文本分类实践（一）：实现一个简单的分类模型 ][Bert_]  
[【NLP】Bert文本分类][NLP_Bert]  
[二分类问题：基于BERT的文本分类实践！附完整代码][BERT]  
[NLP（二十）利用BERT实现文本二分类][NLP_BERT]  
[二分类、多分类与多标签问题的区别及对应损失函数的选择][Link 1]

[watermark_type_d3F5LXplbmhlaQ_shadow_50_text_Q1NETiBA5b-N6ICF44Gu5Lmx5aSq6YOO_size_20_color_FFFFFF_t_70_g_se_x_16]: https://img-blog.csdnimg.cn/7f47d09ce3e644cba047dabeff2365b6.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5b-N6ICF44Gu5Lmx5aSq6YOO,size_20,color_FFFFFF,t_70,g_se,x_16
[Multi-label Text Classification using BERT _ The Mighty Transformer]: https://medium.com/huggingface/multi-label-text-classification-using-bert-the-mighty-transformer-69714fa3fb3d
[https_github.com_huggingface_transformers]: https://github.com/huggingface/transformers
[Bert_fine-tuning]: https://www.jianshu.com/p/bea5abbc2278
[_ BERT fine-tune]: https://www.jianshu.com/p/aa2eff7ec5c1
[Bert_]: https://www.cnblogs.com/qingyao/p/15389307.html
[NLP_Bert]: https://zhuanlan.zhihu.com/p/91533809
[BERT]: https://cloud.tencent.com/developer/article/1601710
[NLP_BERT]: https://segmentfault.com/a/1190000021734797
[Link 1]: https://blog.csdn.net/lyy14011305/article/details/88664518