论文标题
随机反向传播的深入研究
An In-depth Study of Stochastic Backpropagation
论文作者
论文摘要
在本文中，当训练深层神经网络以进行标准图像分类和对象检测任务时，我们提供了对随机反向传播（SBP）的深入研究。在向后传播期间，SBP仅使用特征图的子集来节省GPU内存和计算成本，从而计算梯度。我们将SBP解释为通过执行反向传播辍学来实现随机梯度体面的有效方法，这导致了大量的内存节省和训练过程的速度，对整体模型准确性产生了最小的影响。我们提供了一些很好的实践来应用SBP在培训图像识别模型中，这些模型可以在学习广泛的深层神经网络时采用。关于图像分类和对象检测的实验表明，SBP可以节省多达40％的GPU存储器，而精度降解率少于1％。
In this paper, we provide an in-depth study of Stochastic Backpropagation (SBP) when training deep neural networks for standard image classification and object detection tasks. During backward propagation, SBP calculates the gradients by only using a subset of feature maps to save the GPU memory and computational cost. We interpret SBP as an efficient way to implement stochastic gradient decent by performing backpropagation dropout, which leads to considerable memory saving and training process speedup, with a minimal impact on the overall model accuracy. We offer some good practices to apply SBP in training image recognition models, which can be adopted in learning a wide range of deep neural networks. Experiments on image classification and object detection show that SBP can save up to 40% of GPU memory with less than 1% accuracy degradation.
