论文标题
技术报告 - 使用预读语言模型及时调整的竞争解决方案
Technical Report -- Competition Solution for Prompt Tuning using Pretrained Language Model
论文作者
论文摘要
最近，迅速调整成为大型审计语言模型在特定下游任务上的应用中的热点。关于语言模型作为服务（LMAA），使用无导数优化（DFO）进行黑盒调整提供了一种新颖的方法，可以扩展预验证模型的实际场景，并丰富少量学习的研究。在本报告中，我们在基于LMAA的场景的比赛中介绍了解决方案。我们的解决方案包括对BBTV2的几种修改，包括多个标签单词，选择P0，滚动更新策略，来自MLP分类器的多任务损失，最后使用集合方法进一步提高了概括能力。我们还分享了一些我们尝试过的策略，但在最终提交中没有使用这些策略进行进一步讨论。最后，我们提出了一个有关SNLI数据集以及对结果的影响的问题，以及我们对竞争的担忧。
Prompt tuning recently becomes a hot-spot in the applications of large pretrained language models on specific downstream tasks. Regarding the Language Model as a Service (LMaaS), black-box tuning using derivative-free optimization (DFO) provides a novel approach to expand the practical scenarios of pretrained models and enrich the researches of few-shot learning. In this report, we present our solution in this competition that is based on the LMaaS scenario. Our solution consists of several modifications to BBTv2, including multiple label words, selection of P0, rolling update strategy, multi-task loss from MLP classifier, and finally using the ensemble method to further improve generalization ability. We also shared some strategies that we tried but didn't use in the final submission for further discussion. In the end we raised a question about the SNLI dataset and the impact on the results, as well as our concerns about the competition.
