论文标题
PEA：通过使用渐进合奏激活免费提高Relu网络的性能
PEA: Improving the Performance of ReLU Networks for Free by Using Progressive Ensemble Activations
论文作者
论文摘要
近年来，已经提出了新颖的激活功能来提高神经网络的性能，并且与Relu对应物相比，它们的性能优越。但是，在某些环境中，复杂激活的可用性受到限制，通常只有支持relu。在本文中，我们提出的方法可用于通过在模型训练期间使用这些有效的新型激活来改善Relu网络的性能。更具体地说，我们提出了由relu和这些新型激活之一组成的集合激活。此外，合奏系数既不固定也不是固定的，而是在训练过程中逐渐更新的方式，即到训练结束时，只有RELU激活仍在网络中保持活跃，并且可以删除其他激活。这意味着在推理时间内，网络仅包含RELU激活。我们使用各种紧凑的网络体系结构和各种新颖的激活功能对Imagenet分类任务进行广泛的评估。结果显示0.2-0.8％TOP-1准确性增益，这证实了所提出方法的适用性。此外，我们演示了关于语义分割的建议方法，并在CityScapes数据集上提高了紧凑型分割网络的性能。
In recent years novel activation functions have been proposed to improve the performance of neural networks, and they show superior performance compared to the ReLU counterpart. However, there are environments, where the availability of complex activations is limited, and usually only the ReLU is supported. In this paper we propose methods that can be used to improve the performance of ReLU networks by using these efficient novel activations during model training. More specifically, we propose ensemble activations that are composed of the ReLU and one of these novel activations. Furthermore, the coefficients of the ensemble are neither fixed nor learned, but are progressively updated during the training process in a way that by the end of the training only the ReLU activations remain active in the network and the other activations can be removed. This means that in inference time the network contains ReLU activations only. We perform extensive evaluations on the ImageNet classification task using various compact network architectures and various novel activation functions. Results show 0.2-0.8% top-1 accuracy gain, which confirms the applicability of the proposed methods. Furthermore, we demonstrate the proposed methods on semantic segmentation and we boost the performance of a compact segmentation network by 0.34% mIOU on the Cityscapes dataset.
