论文标题
自然主义变化在目标对话中的影响
Effects of Naturalistic Variation in Goal-Oriented Dialog
论文作者
论文摘要
用于评估端到端神经对话系统性能的现有基准缺乏关键组成部分：人类对话中存在的自然变化。大多数数据集都是通过众包构建的，在众包中，人群工人在执行用户/代理的角色时遵循固定的指令模板。这会导致直截了当的，有些例行的，并且主要是无故障的对话，因为人群工作人员认为并不代表自然而然地与真实用户发生的全部动作。在这项工作中，我们研究了自然主义变化对两个面向目标数据集的影响：BABI对话任务和Stanford多域数据集（SMD）。我们还通过引入用户的自然变化来为两个数据集提出新的，更有效的测试床。我们观察到，最近最新的端到端神经方法（例如BossNet和GLMP）在两个数据集上的性能显着下降（SMD上的Ent。F1超过60％，每二键准确性的85％，每二键准确性85％）。
Existing benchmarks used to evaluate the performance of end-to-end neural dialog systems lack a key component: natural variation present in human conversations. Most datasets are constructed through crowdsourcing, where the crowd workers follow a fixed template of instructions while enacting the role of a user/agent. This results in straight-forward, somewhat routine, and mostly trouble-free conversations, as crowd workers do not think to represent the full range of actions that occur naturally with real users. In this work, we investigate the impact of naturalistic variation on two goal-oriented datasets: bAbI dialog task and Stanford Multi-Domain Dataset (SMD). We also propose new and more effective testbeds for both datasets, by introducing naturalistic variation by the user. We observe that there is a significant drop in performance (more than 60% in Ent. F1 on SMD and 85% in per-dialog accuracy on bAbI task) of recent state-of-the-art end-to-end neural methods such as BossNet and GLMP on both datasets.
