论文标题
托德·伯特（Tod-Bert）：针对任务对话的预训练的自然语言理解
TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue
论文作者
论文摘要
一般文本和面向任务的对话之间语言模式的潜在差异使现有的预训练的语言模型在实践中的有用程度降低。在这项工作中，我们统一了九个人类人类和多转弯的对话数据集用于语言建模。为了在预训练期间更好地建模对话行为，我们将用户和系统令牌纳入蒙版语言建模中。我们提出了一个对比目标函数，以模拟响应选择任务。我们预先训练的以任务为导向的对话Bert（Tod-Bert）在四个下游以任务为导向的对话应用程序上优于Bert，例如Bert，包括意图识别，对话状态跟踪，对话ACT ACT预测和响应选择。我们还表明，托德·伯特（Tod-Bert）具有更强的射击能力，可以减轻以任务为导向对话的数据稀缺问题。
The underlying difference of linguistic patterns between general text and task-oriented dialogue makes existing pre-trained language models less useful in practice. In this work, we unify nine human-human and multi-turn task-oriented dialogue datasets for language modeling. To better model dialogue behavior during pre-training, we incorporate user and system tokens into the masked language modeling. We propose a contrastive objective function to simulate the response selection task. Our pre-trained task-oriented dialogue BERT (TOD-BERT) outperforms strong baselines like BERT on four downstream task-oriented dialogue applications, including intention recognition, dialogue state tracking, dialogue act prediction, and response selection. We also show that TOD-BERT has a stronger few-shot ability that can mitigate the data scarcity problem for task-oriented dialogue.
