SentiBERT: Pre-training Language Model Combining Sentiment Information

SentiBERT: Pre-training Language Model Combining Sentiment Information

Blog Article

Pre-training language models on large-scale unsupervised corpus are attracting the attention of researchers in the field of natural language processing.The existing model mainly extracts the semantic and structural features of the text in the pre-training stage.Aiming at sentiment task and complex emotional features, a pre-training method focusing on learning sentiment features is proposed 27" Warming Drawer on the basis of the latest pre-training language model BERT(bidirectional encoder representations from transformers).In the further pre-training stage, this paper improves pre-training task of BERT with the help of sentiment dictionary.At the same time, this paper uses context-based word sentiment prediction task to classify the sentiment of masked words to acquire the textual representation biased towards sentiment features.

Finally, fine-tuning is performed on a small labeled data sets.Experimental results show that, compared with the original BERT model, the accuracy RAPID MASS ICE CREAM COOKIES of sentiment tasks can be improved by 1 percentage point.More advanced results can be achieved at small training sets.