论文标题
有效的对抗性训练，可与稳健的早鸟门票
Efficient Adversarial Training with Robust Early-Bird Tickets
论文作者
论文摘要
对抗训练是提高预训练语言模型（PLM）鲁棒性的最强大方法之一。但是，这种方法通常比传统的微调更昂贵，因为有必要通过梯度下降产生对抗性示例。深入研究对抗性训练的优化过程，我们发现在早期训练阶段出现了强大的连接模式（通常为$ 0.15 \ sim0.3 $ epochs），在参数收敛之前。受这一发现的启发，我们挖掘出强大的早期票票（即子网）来开发一种有效的对抗训练方法：（1）在早期搜索具有结构性稀疏性的健壮门票； （2）在剩余时间内微调可靠的门票。为了尽早提取健壮的门票，我们设计了一个票务收敛度量，以自动终止搜索过程。实验表明，与最有竞争力的最先进的对抗训练方法相比，提出的有效的对抗训练方法可以实现高达$ 7 \ times \ sim 13 \ sim 13 \ times $训练速度，同时保持可比性甚至更好的鲁棒性。
Adversarial training is one of the most powerful methods to improve the robustness of pre-trained language models (PLMs). However, this approach is typically more expensive than traditional fine-tuning because of the necessity to generate adversarial examples via gradient descent. Delving into the optimization process of adversarial training, we find that robust connectivity patterns emerge in the early training phase (typically $0.15\sim0.3$ epochs), far before parameters converge. Inspired by this finding, we dig out robust early-bird tickets (i.e., subnetworks) to develop an efficient adversarial training method: (1) searching for robust tickets with structured sparsity in the early stage; (2) fine-tuning robust tickets in the remaining time. To extract the robust tickets as early as possible, we design a ticket convergence metric to automatically terminate the searching process. Experiments show that the proposed efficient adversarial training method can achieve up to $7\times \sim 13 \times$ training speedups while maintaining comparable or even better robustness compared to the most competitive state-of-the-art adversarial training methods.
