论文标题
颂歌
An Ode to an ODE
论文作者
论文摘要
我们提出了一种称为Odetoode的神经ode算法的新范式，其中主流的时间依赖性参数根据正交组O（d）上的矩阵流而发展。这两个流的嵌套系统，其中参数流的限制在紧凑的歧管上，可提供训练的稳定性和有效性，并证明解决了与训练深层神经网络体系结构（例如神经odes）的梯度消失探索问题。因此，它通过与以前的SOTA基线相比，在培训强化学习政策以及在监督的学习环境中培训强化学习政策的示例中，它导致了更好的下游模型。我们为我们提出的机制提供了强大的收敛结果，该机制独立于网络的深度，支持我们的经验研究。我们的结果表明，深神经网络理论与紧凑型歧管上的基质流动之间有着有趣的联系。
We present a new paradigm for Neural ODE algorithms, called ODEtoODE, where time-dependent parameters of the main flow evolve according to a matrix flow on the orthogonal group O(d). This nested system of two flows, where the parameter-flow is constrained to lie on the compact manifold, provides stability and effectiveness of training and provably solves the gradient vanishing-explosion problem which is intrinsically related to training deep neural network architectures such as Neural ODEs. Consequently, it leads to better downstream models, as we show on the example of training reinforcement learning policies with evolution strategies, and in the supervised learning setting, by comparing with previous SOTA baselines. We provide strong convergence results for our proposed mechanism that are independent of the depth of the network, supporting our empirical studies. Our results show an intriguing connection between the theory of deep neural networks and the field of matrix flows on compact manifolds.
