论文标题
有效调谐参数是任务嵌入
Efficiently Tuned Parameters are Task Embeddings
论文作者
论文摘要
使用正确选择的源数据集，中级任务转移可以使广泛的NLP任务受益。但是，在计算上尝试所有中间转移组合，这是不可行的，这使得选择有用的源任务是一个具有挑战性的问题。在本文中，我们预计在参数效率调整方法中更新的特定任务参数可能会编码特定于任务的信息。因此，此类参数可以预测任务间的可传递性。因此，我们建议利用这些有效调整的参数作为现成的任务嵌入，以有效地选择用于中间任务传输的源数据集。我们尝试11个文本分类任务和11个问答任务。实验结果表明，我们的方法可以始终超过现有的任务间可传递性预测方法，同时在概念上简单且计算上有效。我们的分析还表明，有效调谐参数在可传递性预测上的能力与他们的任务性能相关。这使我们可以将早期检查点的参数用作任务嵌入，以进一步提高效率。
Intermediate-task transfer can benefit a wide range of NLP tasks with properly selected source datasets. However, it is computationally infeasible to experiment with all intermediate transfer combinations, making choosing a useful source task a challenging problem. In this paper, we anticipate that task-specific parameters updated in parameter-efficient tuning methods are likely to encode task-specific information. Therefore, such parameters can be predictive for inter-task transferability. Thus, we propose to exploit these efficiently tuned parameters as off-the-shelf task embeddings for the efficient selection of source datasets for intermediate-task transfer. We experiment with 11 text classification tasks and 11 question answering tasks. Experimental results show that our approach can consistently outperform existing inter-task transferability prediction methods while being conceptually simple and computationally efficient. Our analysis also reveals that the ability of efficiently tuned parameters on transferability prediction is disentangled with their in-task performance. This allows us to use parameters from early checkpoints as task embeddings to further improve efficiency.
