论文标题

阿波罗:一种简单的方法,用于自适应训练语言模型的逻辑推理

APOLLO: A Simple Approach for Adaptive Pretraining of Language Models for Logical Reasoning

论文作者

Sanyal, Soumya, Xu, Yichong, Wang, Shuohang, Yang, Ziyi, Pryzant, Reid, Yu, Wenhao, Zhu, Chenguang, Ren, Xiang

论文摘要

文本的逻辑推理是一个重要的能力,需要了解文本中存在的信息,它们的互连,然后通过它们推理来推断新结论。提高语言模型的逻辑推理能力的先验工作需要对培训数据进行复杂的处理(例如,将符号知识与文本保持一致),产生特定于任务的数据增强解决方案,从而限制了一般逻辑推理技能的学习。在这项工作中,我们提出了一种自适应的语言模型Apollo,它提高了逻辑推理能力。我们根据一组逻辑推理关键字选择Wikipedia的子集,以继续预测语言模型。我们使用两个自我监督的损失功能:经过修改的掩盖语言建模损失,只有特定的言论单词(可能比基本语言理解)需要更多的推理,而被掩盖了,而句子级别的分类损失则教导模型区分零件和矛盾的句子类型。拟议的培训范式既简单又独立于任务格式。我们通过将阿波罗与两个逻辑推理数据集上的先验基准进行比较来证明阿波罗的有效性。 Apollo在LogiQa上的Reclor上表现出色,并且胜过基线。该代码库已公开可用。

Logical reasoning of text is an important ability that requires understanding the information present in the text, their interconnections, and then reasoning through them to infer new conclusions. Prior works on improving the logical reasoning ability of language models require complex processing of training data (e.g., aligning symbolic knowledge to text), yielding task-specific data augmentation solutions that restrict the learning of general logical reasoning skills. In this work, we propose APOLLO, an adaptively pretrained language model that has improved logical reasoning abilities. We select a subset of Wikipedia, based on a set of logical inference keywords, for continued pretraining of a language model. We use two self-supervised loss functions: a modified masked language modeling loss where only specific parts-of-speech words, that would likely require more reasoning than basic language understanding, are masked, and a sentence-level classification loss that teaches the model to distinguish between entailment and contradiction types of sentences. The proposed training paradigm is both simple and independent of task formats. We demonstrate the effectiveness of APOLLO by comparing it with prior baselines on two logical reasoning datasets. APOLLO performs comparably on ReClor and outperforms baselines on LogiQA. The code base has been made publicly available.

扫码加入交流群

加入微信交流群

微信交流群二维码

扫码加入学术交流群,获取更多资源