论文标题
LERT：语言动机的预训练的语言模型
LERT: A Linguistically-motivated Pre-trained Language Model
论文作者
论文摘要
预训练的语言模型（PLM）已成为自然语言处理领域的代表性基础模型。大多数PLM都接受了文本表面形式上的语言不可屈服的预训练任务，例如蒙版语言模型（MLM）。为了进一步增强PLM的能力，以更丰富的语言特征，我们旨在提出一种简单但有效的方法来学习预训练的语言模型的语言特征。我们提出了LERT是一种预先训练的语言模型，该模型使用语言知名的预训练（LIP）策略对三种类型的语言特征以及原始的MLM预训练任务进行了培训。我们对十项中国NLU任务进行了广泛的实验，实验结果表明，LERT可以比各种类似的基线带来显着改善。此外，我们还在各个语言方面进行了分析实验，结果证明了LERT的设计是有效和有效的。资源可在https://github.com/ymcui/lert上找到
Pre-trained Language Model (PLM) has become a representative foundation model in the natural language processing field. Most PLMs are trained with linguistic-agnostic pre-training tasks on the surface form of the text, such as the masked language model (MLM). To further empower the PLMs with richer linguistic features, in this paper, we aim to propose a simple but effective way to learn linguistic features for pre-trained language models. We propose LERT, a pre-trained language model that is trained on three types of linguistic features along with the original MLM pre-training task, using a linguistically-informed pre-training (LIP) strategy. We carried out extensive experiments on ten Chinese NLU tasks, and the experimental results show that LERT could bring significant improvements over various comparable baselines. Furthermore, we also conduct analytical experiments in various linguistic aspects, and the results prove that the design of LERT is valid and effective. Resources are available at https://github.com/ymcui/LERT
