论文标题
及时调整判别性预训练的语言模型
Prompt Tuning for Discriminative Pre-trained Language Models
论文作者
论文摘要
最近的作品显示了迅速调整自然语言处理（NLP）任务的迅速调整的有希望的结果。但是，据我们所知，现有的作品着重于迅速调整生成PLM，这些生成PLM经过预训练以生成目标令牌，例如Bert。仍然未知是否以及如何有效地调整伊利特拉（Electra）的歧视性PLM。在这项工作中，我们介绍了DPT，这是第一个歧视性PLM的及时调整框架，该框架将NLP任务重新制定为歧视性语言建模问题。关于文本分类和问题答案的全面实验表明，与香草微调相比，DPT的性能明显更高，并且还可以防止在全设定和低资产设置中调整大型PLM的不稳定问题。本文的源代码和实验详细信息可以从https://github.com/thunlp/dpt获得。
Recent works have shown promising results of prompt tuning in stimulating pre-trained language models (PLMs) for natural language processing (NLP) tasks. However, to the best of our knowledge, existing works focus on prompt-tuning generative PLMs that are pre-trained to generate target tokens, such as BERT. It is still unknown whether and how discriminative PLMs, e.g., ELECTRA, can be effectively prompt-tuned. In this work, we present DPT, the first prompt tuning framework for discriminative PLMs, which reformulates NLP tasks into a discriminative language modeling problem. Comprehensive experiments on text classification and question answering show that, compared with vanilla fine-tuning, DPT achieves significantly higher performance, and also prevents the unstable problem in tuning large PLMs in both full-set and low-resource settings. The source code and experiment details of this paper can be obtained from https://github.com/thunlp/DPT.
