论文标题
快捷条件：一种视觉分析方法，用于探索自然语言中的快捷方式理解数据集
ShortcutLens: A Visual Analytics Approach for Exploring Shortcuts in Natural Language Understanding Dataset
论文作者
论文摘要
基准数据集在评估自然语言理解（NLU）模型中起着重要作用。但是，快捷方式 - 基准数据集中有害的偏见 - 可能会损害基准数据集在揭示模型的实际功能中的有效性。由于快捷方式的覆盖范围，生产力和语义含义各不相同，因此NLU专家在创建基准数据集时系统地理解和避免它们是一项挑战。在本文中，我们开发了一个视觉分析系统，即短速度，以帮助NLU专家探索NLU基准数据集中的快捷方式。该系统允许用户对快捷方式进行多层探索。具体来说，统计信息可以帮助用户掌握统计数据，例如基准数据集中快捷方式的覆盖范围和生产力。模板视图采用层次和可解释的模板来汇总不同类型的快捷方式。实例视图允许用户检查快捷方式涵盖的相应实例。我们进行案例研究和专家访谈，以评估系统的有效性和可用性。结果表明，快捷方式支持用户通过快捷方式更好地了解基准数据集问题，从而激发了他们创建具有挑战性且相关的基准数据集。
Benchmark datasets play an important role in evaluating Natural Language Understanding (NLU) models. However, shortcuts -- unwanted biases in the benchmark datasets -- can damage the effectiveness of benchmark datasets in revealing models' real capabilities. Since shortcuts vary in coverage, productivity, and semantic meaning, it is challenging for NLU experts to systematically understand and avoid them when creating benchmark datasets. In this paper, we develop a visual analytics system, ShortcutLens, to help NLU experts explore shortcuts in NLU benchmark datasets. The system allows users to conduct multi-level exploration of shortcuts. Specifically, Statistics View helps users grasp the statistics such as coverage and productivity of shortcuts in the benchmark dataset. Template View employs hierarchical and interpretable templates to summarize different types of shortcuts. Instance View allows users to check the corresponding instances covered by the shortcuts. We conduct case studies and expert interviews to evaluate the effectiveness and usability of the system. The results demonstrate that ShortcutLens supports users in gaining a better understanding of benchmark dataset issues through shortcuts, inspiring them to create challenging and pertinent benchmark datasets.
