论文标题
超越反向传播：通过隐式分化和均衡传播的双重优化
Beyond backpropagation: bilevel optimization through implicit differentiation and equilibrium propagation
论文作者
论文摘要
本文回顾了基于梯度的技术来解决二聚体优化问题。二重性优化是一种构造学习系统通过最小化的数量进行隐式定义的系统的一般方法。与此类系统的明确定义相比，该表征可以应用于神经网络，优化器，算法求解器甚至物理系统，并允许更大的建模灵活性。在这里，我们专注于解决此类问题的基于梯度的方法。我们将它们分为两类：植根于隐式分化的类别，以及那些利用平衡传播定理的分化。我们介绍了这种方法背后的数学基础，详细介绍了梯度估计算法并比较不同方法的竞争优势。
This paper reviews gradient-based techniques to solve bilevel optimization problems. Bilevel optimization is a general way to frame the learning of systems that are implicitly defined through a quantity that they minimize. This characterization can be applied to neural networks, optimizers, algorithmic solvers and even physical systems, and allows for greater modeling flexibility compared to an explicit definition of such systems. Here we focus on gradient-based approaches that solve such problems. We distinguish them in two categories: those rooted in implicit differentiation, and those that leverage the equilibrium propagation theorem. We present the mathematical foundations that are behind such methods, introduce the gradient-estimation algorithms in detail and compare the competitive advantages of the different approaches.
