论文标题
通过多视图压缩表示形式进行稳健的低资源微调
Towards Robust Low-Resource Fine-Tuning with Multi-View Compressed Representations
论文作者
论文摘要
由于参数大量，预审前的语言模型（PLM）很容易在低资源方案中过度拟合。在这项工作中，我们提出了一种新颖的方法，该方法在PLM的隐藏表示状态下运行以减少过度拟合。在微调过程中，我们的方法在PLM的隐藏层之间插入随机自动编码器，该自动编码器将激活从以前的层转换为多视图压缩表示，然后再将其馈入上层。微调后，自动编码器会插入，因此我们的方法不会添加额外的参数或增加推断期间的计算成本。我们的方法证明了各种序列和令牌级别的低资源NLP任务的有希望的性能提高。
Due to the huge amount of parameters, fine-tuning of pretrained language models (PLMs) is prone to overfitting in the low resource scenarios. In this work, we present a novel method that operates on the hidden representations of a PLM to reduce overfitting. During fine-tuning, our method inserts random autoencoders between the hidden layers of a PLM, which transform activations from the previous layers into multi-view compressed representations before feeding them into the upper layers. The autoencoders are plugged out after fine-tuning, so our method does not add extra parameters or increase computation cost during inference. Our method demonstrates promising performance improvement across a wide range of sequence- and token-level low-resource NLP tasks.
