论文标题
深NLP模型的系统性，组成性和传递性：变质测试的观点
Systematicity, Compositionality and Transitivity of Deep NLP Models: a Metamorphic Testing Perspective
论文作者
论文摘要
变质测试最近已用于检查神经NLP模型的安全性。它的主要优点是它不依靠地面真理来生成测试案例。但是，现有的研究主要与稳健性的变质关系有关，从而限制了他们可以测试的语言特性范围。我们提出了三个新的变质关系类别，这些关系涉及系统性，组成性和传递性的特性。与鲁棒性不同，我们的关系是在多个源输入上定义的，从而增加了我们可以通过多项式因素产生的测试用例数量。有了它们，我们测试了最先进的NLP模型的内部一致性，并表明它们并不总是根据其预期的语言特性行为。最后，我们介绍了一种新颖的图形符号，该图形有效地总结了变质关系的内部结构。
Metamorphic testing has recently been used to check the safety of neural NLP models. Its main advantage is that it does not rely on a ground truth to generate test cases. However, existing studies are mostly concerned with robustness-like metamorphic relations, limiting the scope of linguistic properties they can test. We propose three new classes of metamorphic relations, which address the properties of systematicity, compositionality and transitivity. Unlike robustness, our relations are defined over multiple source inputs, thus increasing the number of test cases that we can produce by a polynomial factor. With them, we test the internal consistency of state-of-the-art NLP models, and show that they do not always behave according to their expected linguistic properties. Lastly, we introduce a novel graphical notation that efficiently summarises the inner structure of metamorphic relations.
