论文标题
有效的自然梯度下降方法，用于大规模PDE的优化问题
Efficient Natural Gradient Descent Methods for Large-Scale PDE-Based Optimization Problems
论文作者
论文摘要
我们提出了有效的数值方案，用于实施自然梯度下降（NGD），以适用于广泛的度量空间，并应用于基于PDE的优化问题。我们的技术代表了自然梯度方向作为解决标准最小二乘问题的解决方案。因此，我们没有直接计算，存储或反转信息矩阵，而是采用了数值线性代数的有效方法。我们将两个方案视为雅各布式（即状态变量相对于参数的衍生物）是通过约束明确知道或隐式给出的。因此，我们可以可靠地计算出几个天然NGD，以用于大规模参数空间。特别是，我们能够在数千个维度上计算Wasserstein NGD，这被认为是遥不可及的。最后，我们的数值结果阐明了基于非convex优化问题中不同度量空间的标准梯度下降与各种NGD方法之间的定性差异。
We propose efficient numerical schemes for implementing the natural gradient descent (NGD) for a broad range of metric spaces with applications to PDE-based optimization problems. Our technique represents the natural gradient direction as a solution to a standard least-squares problem. Hence, instead of calculating, storing, or inverting the information matrix directly, we apply efficient methods from numerical linear algebra. We treat both scenarios where the Jacobian, i.e., the derivative of the state variable with respect to the parameter, is either explicitly known or implicitly given through constraints. We can thus reliably compute several natural NGDs for a large-scale parameter space. In particular, we are able to compute Wasserstein NGD in thousands of dimensions, which was believed to be out of reach. Finally, our numerical results shed light on the qualitative differences between the standard gradient descent and various NGD methods based on different metric spaces in nonconvex optimization problems.
