论文标题
测试语言模型解释象征性语言的能力
Testing the Ability of Language Models to Interpret Figurative Language
论文作者
论文摘要
象征性和隐喻性语言在话语中很普遍，象征性的表达在交流和认知中起着重要作用。但是，比喻语言在NLP中是一个相对研究的领域，它仍然是一个开放的问题，在多大程度上，现代语言模型可以解释非文字短语。为了解决这个问题，我们介绍了FIG-QA，这是Winograd风格的非文字理解任务，该任务包括正确解释具有不同含义的配对的比喻短语。我们在此任务上评估了几种最先进的语言模型的性能，并发现语言模型在偶然的情况下达到了很大的效果，但它们仍然没有人类的绩效，尤其是在零或几次的设置中。这表明需要进一步的工作来提高语言模型的非文字推理能力。
Figurative and metaphorical language are commonplace in discourse, and figurative expressions play an important role in communication and cognition. However, figurative language has been a relatively under-studied area in NLP, and it remains an open question to what extent modern language models can interpret nonliteral phrases. To address this question, we introduce Fig-QA, a Winograd-style nonliteral language understanding task consisting of correctly interpreting paired figurative phrases with divergent meanings. We evaluate the performance of several state-of-the-art language models on this task, and find that although language models achieve performance significantly over chance, they still fall short of human performance, particularly in zero- or few-shot settings. This suggests that further work is needed to improve the nonliteral reasoning capabilities of language models.
