论文标题
基于语义的预训练以进行对话理解
Semantic-based Pre-training for Dialogue Understanding
论文作者
论文摘要
预训练的语言模型在对话任务上取得了长足的进步。但是，这些模型通常是在表面对话文本上训练的，因此在理解对话环境的主要语义含义方面被证明是弱的。我们研究抽象含义表示（AMR）作为训练前模型的明确语义知识，以捕获预训练期间对话中的核心语义信息。特别是，我们提出了一个基于语义的前训练框架，该框架通过三个任务来扩展标准的预训练框架（Devlin等，2019）。关于聊天聊天和面向任务的对话的理解的实验表明了我们的模型的优势。据我们所知，我们是第一个利用深层语义表示进行对话预训练的人。
Pre-trained language models have made great progress on dialogue tasks. However, these models are typically trained on surface dialogue text, thus are proven to be weak in understanding the main semantic meaning of a dialogue context. We investigate Abstract Meaning Representation (AMR) as explicit semantic knowledge for pre-training models to capture the core semantic information in dialogues during pre-training. In particular, we propose a semantic-based pre-training framework that extends the standard pre-training framework (Devlin et al., 2019) by three tasks for learning 1) core semantic units, 2) semantic relations and 3) the overall semantic representation according to AMR graphs. Experiments on the understanding of both chit-chats and task-oriented dialogues show the superiority of our model. To our knowledge, we are the first to leverage a deep semantic representation for dialogue pre-training.
