论文标题
Relu网络中线性区域的切线空间灵敏度和分布
Tangent Space Sensitivity and Distribution of Linear Regions in ReLU Networks
论文作者
论文摘要
最近的文章表明，深层神经网络是各种学习问题的有效模型。但是，它们通常对独立观察者无法检测到的各种变化高度敏感。由于我们对具有传统概括范围的深神经网络的理解仍然不完整，因此，在特定状态下发生了一些小变化的情况下，有几种措施捕获了模型的行为。在本文中，我们考虑在切线空间中的对抗稳定性，并提出切线灵敏度以表征稳定性。我们关注一种特定的稳定性，这些稳定性是由没有已知标签的个别示例引起的参数变化。我们得出了几个易于计算的界限和经验度量，用于馈电完全连接的relu（整流线性单元）网络，并将切线灵敏度连接到网络实现的输入空间中激活区域的分布。我们的实验表明，即使是简单的界限和措施也与经验概括差距有关。
Recent articles indicate that deep neural networks are efficient models for various learning problems. However they are often highly sensitive to various changes that cannot be detected by an independent observer. As our understanding of deep neural networks with traditional generalization bounds still remains incomplete, there are several measures which capture the behaviour of the model in case of small changes at a specific state. In this paper we consider adversarial stability in the tangent space and suggest tangent sensitivity in order to characterize stability. We focus on a particular kind of stability with respect to changes in parameters that are induced by individual examples without known labels. We derive several easily computable bounds and empirical measures for feed-forward fully connected ReLU (Rectified Linear Unit) networks and connect tangent sensitivity to the distribution of the activation regions in the input space realized by the network. Our experiments suggest that even simple bounds and measures are associated with the empirical generalization gap.
