论文标题
在一个隐藏的稀疏网络及以后的景观上
On the Landscape of One-hidden-layer Sparse Networks and Beyond
论文作者
论文摘要
与密集的网络相比，由于其尺寸较小，稀疏的神经网络已获得越来越多的兴趣。然而，大多数现有关于神经网络理论的作品都集中在密集的神经网络上，对稀疏网络的理解非常有限。在本文中，我们研究了一个隐藏的稀疏网络的损失格局。首先，我们考虑具有致密最终层的稀疏网络。我们表明，线性网络在特殊的稀疏结构下不可能有虚假的山谷，而非线性网络也可以承认在宽的最后一层下没有虚假的山谷。其次，我们发现，对于稀疏的最后一层稀疏网络，可以存在虚假的山谷和虚假的极小山脉。这与在轻度假设下没有虚假山谷的广泛密集网络不同。
Sparse neural networks have received increasing interest due to their small size compared to dense networks. Nevertheless, most existing works on neural network theory have focused on dense neural networks, and the understanding of sparse networks is very limited. In this paper, we study the loss landscape of one-hidden-layer sparse networks. First, we consider sparse networks with a dense final layer. We show that linear networks can have no spurious valleys under special sparse structures, and non-linear networks could also admit no spurious valleys under a wide final layer. Second, we discover that spurious valleys and spurious minima can exist for wide sparse networks with a sparse final layer. This is different from wide dense networks which do not have spurious valleys under mild assumptions.
