论文标题
具有固定张量训练等级的张量的二阶优化
Second-order optimization for tensors with fixed tensor-train rank
论文作者
论文摘要
与不同格式相关的张量有几种不同的“低等级”概念。其中，张量列车（TT）格式特别适合高阶张量，因为它规避了维度的诅咒：某些高维应用的可观属性。通常是在固定（和低）TT等级的张量张量的优化之类的应用程序中建模的应用程序通常很方便。那组是一个平滑的多种多样。利用这一事实，其他人表明，黎曼优化技术可以在诸如PDES的张量完成和特殊的大规模线性系统等任务上表现出色。然而，到目前为止，这些优化技术仅限于一阶方法，这可能是由于推导了Riemannian Hessian精确表达的技术障碍。在本文中，我们得出了一种公式和有效的算法来计算此歧管上的Riemannian Hessian。这使我们能够实现二阶优化算法（即Riemannian Trust-Region方法），并分析优化问题的条件在固定的TT等级歧管上。在感兴趣的设置中，与一阶方法和交替的最小二乘（ALS）相比，我们显示出张量完成的优化性能的提高。我们的工作可能在培训具有张量层的神经网络中的应用。我们的代码是免费的。
There are several different notions of "low rank" for tensors, associated to different formats. Among them, the Tensor Train (TT) format is particularly well suited for tensors of high order, as it circumvents the curse of dimensionality: an appreciable property for certain high-dimensional applications. It is often convenient to model such applications as optimization over the set of tensors with fixed (and low) TT rank. That set is a smooth manifold. Exploiting this fact, others have shown that Riemannian optimization techniques can perform particularly well on tasks such as tensor completion and special large-scale linear systems from PDEs. So far, however, these optimization techniques have been limited to first-order methods, likely because of the technical hurdles in deriving exact expressions for the Riemannian Hessian. In this paper, we derive a formula and efficient algorithm to compute the Riemannian Hessian on this manifold. This allows us to implement second-order optimization algorithms (namely, the Riemannian trust-region method) and to analyze the conditioning of optimization problems over the fixed TT rank manifold. In settings of interest, we show improved optimization performance on tensor completion compared to first-order methods and alternating least squares (ALS). Our work could have applications in training of neural networks with tensor layers. Our code is freely available.
