论文标题
过失：在自然语言理解任务中，句子中单词的顺序顺序有多重要？
Out of Order: How Important Is The Sequential Order of Words in a Sentence in Natural Language Understanding Tasks?
论文作者
论文摘要
最先进的自然语言理解模型是否关心单词顺序 - 序列最重要的特征之一？并非总是！我们发现，经过许多胶水任务训练的基于BERT的分类器的正确预测的75％至90％，在输入单词被随机改组后保持恒定。尽管Bert嵌入是著名的上下文，但即使在单词的上下文被改组之后，每个单词对下游任务的贡献几乎没有改变。基于BERT的模型能够利用表面提示（例如，情感分析中关键字的情感；或自然语言推论中序列输入之间的单词相似性），以在随机订单中排列令牌时做出正确的决策。鼓励分类器捕获单词订单信息可改善大多数胶水任务，小队2.0和示例外的性能。我们的工作表明，许多胶水任务并不是理解句子含义的机器。
Do state-of-the-art natural language understanding models care about word order - one of the most important characteristics of a sequence? Not always! We found 75% to 90% of the correct predictions of BERT-based classifiers, trained on many GLUE tasks, remain constant after input words are randomly shuffled. Despite BERT embeddings are famously contextual, the contribution of each individual word to downstream tasks is almost unchanged even after the word's context is shuffled. BERT-based models are able to exploit superficial cues (e.g. the sentiment of keywords in sentiment analysis; or the word-wise similarity between sequence-pair inputs in natural language inference) to make correct decisions when tokens are arranged in random orders. Encouraging classifiers to capture word order information improves the performance on most GLUE tasks, SQuAD 2.0 and out-of-samples. Our work suggests that many GLUE tasks are not challenging machines to understand the meaning of a sentence.
