论文标题
SMG：一种基于动量的基于梯度的方法
SMG: A Shuffling Gradient-Based Method with Momentum
论文作者
论文摘要
我们结合了两种用于机器学习的优化的高级想法：改组策略和动量技术，以开发一种基于动量的基于动量的基于梯度的方法，用于非convex有限和升级的优化问题。尽管我们的方法灵感来自动量技术，但其更新与现有基于动量的方法根本不同。我们在标准假设（即$ l $ -smoothness and Bounded差异）下使用恒定或降低的学习率来建立任何改组策略的最先进的SMG融合率。固定改组策略时，我们会开发另一种类似于现有动量方法的新算法，并在$ l $ smoothness和有界梯度假设下证明了该算法的相同收敛速率。我们通过标准数据集的数值模拟演示了我们的算法，并将其与现有的改组方法进行比较。我们的测试表明新算法的性能令人鼓舞。
We combine two advanced ideas widely used in optimization for machine learning: shuffling strategy and momentum technique to develop a novel shuffling gradient-based method with momentum, coined Shuffling Momentum Gradient (SMG), for non-convex finite-sum optimization problems. While our method is inspired by momentum techniques, its update is fundamentally different from existing momentum-based methods. We establish state-of-the-art convergence rates of SMG for any shuffling strategy using either constant or diminishing learning rate under standard assumptions (i.e.$L$-smoothness and bounded variance). When the shuffling strategy is fixed, we develop another new algorithm that is similar to existing momentum methods, and prove the same convergence rates for this algorithm under the $L$-smoothness and bounded gradient assumptions. We demonstrate our algorithms via numerical simulations on standard datasets and compare them with existing shuffling methods. Our tests have shown encouraging performance of the new algorithms.
