论文标题

具有不同隐私的大型卷积神经网络的可扩展有效培训

Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacy

论文作者

Bu, Zhiqi, Mao, Jialin, Xu, Shiyun

论文摘要

大型卷积神经网络(CNN)可能很难在差异私有(DP)方向上进行训练,因为优化算法需要计算昂贵的操作,称为每样本梯度剪辑。我们提出了对卷积层的这种剪辑的有效且可扩展的实施,称为混合的幽灵剪裁,从而在不影响准确性的情况下大大简化了私人培训的情况。通过对混合幽灵剪辑和现有的DP培训算法进行的首次复杂性分析对效率的提高进行了严格的研究。 关于视力分类任务的广泛实验,具有大型重新连接,VGG和视觉变压器,表明与混合幽灵剪裁的DP培训增加了$ 1 \ sim 10 \%$内存开销,$ <2 \ times $ $ slowdown in Stardard非私人培训。具体来说,当在CIFAR10上培训VGG19时,混合的幽灵剪裁的价格是$ 3 \ times $ $ $ $比最先进的Opa​​cus库,$ 18 \ times $ $最大批处理大小。为了强调有效的DP培训对卷积层的重要性,我们使用BEIT实现了CIFAR10上的96.7 \%精度和CIFAR100的83.0 \%\%的精度,而先前的最佳结果分别为94.8%\%\%和67.4 \%。我们打开隐私引擎(\ url {https://github.com/woodyx218/private_vision}),该引擎通过几行代码来实现CNN的DP培训。

Large convolutional neural networks (CNN) can be difficult to train in the differentially private (DP) regime, since the optimization algorithms require a computationally expensive operation, known as the per-sample gradient clipping. We propose an efficient and scalable implementation of this clipping on convolutional layers, termed as the mixed ghost clipping, that significantly eases the private training in terms of both time and space complexities, without affecting the accuracy. The improvement in efficiency is rigorously studied through the first complexity analysis for the mixed ghost clipping and existing DP training algorithms. Extensive experiments on vision classification tasks, with large ResNet, VGG, and Vision Transformers, demonstrate that DP training with mixed ghost clipping adds $1\sim 10\%$ memory overhead and $<2\times$ slowdown to the standard non-private training. Specifically, when training VGG19 on CIFAR10, the mixed ghost clipping is $3\times$ faster than state-of-the-art Opacus library with $18\times$ larger maximum batch size. To emphasize the significance of efficient DP training on convolutional layers, we achieve 96.7\% accuracy on CIFAR10 and 83.0\% on CIFAR100 at $ε=1$ using BEiT, while the previous best results are 94.8\% and 67.4\%, respectively. We open-source a privacy engine (\url{https://github.com/woodyx218/private_vision}) that implements DP training of CNN with a few lines of code.

扫码加入交流群

加入微信交流群

微信交流群二维码

扫码加入学术交流群,获取更多资源