论文标题
深度无监督的对比学习中端到端培训的融合
Convergence of End-to-End Training in Deep Unsupervised Contrastive Learning
论文作者
论文摘要
在最新研究中，无监督的对比学习引起了人们的关注，并被证明是一种从未标记的数据学习表示形式的有力方法。但是，对于此框架，很少有理论分析。在本文中，我们研究了深度无监督的对比学习的优化。我们证明，通过应用端到端训练，同时更新了两个深度参数过度的神经网络，可以找到一个近似的固定解决方案，以实现非凸面对比损失。该结果本质上与监督环境中现有的过度参数分析有所不同，因为与学习特定的目标功能相反，无监督的对比学习试图将未标记的数据分布编码到神经网络中，而神经网络通常没有最佳解决方案。我们的分析提供了对这些无监督预处理方法实际成功的理论见解。
Unsupervised contrastive learning has gained increasing attention in the latest research and has proven to be a powerful method for learning representations from unlabeled data. However, little theoretical analysis was known for this framework. In this paper, we study the optimization of deep unsupervised contrastive learning. We prove that, by applying end-to-end training that simultaneously updates two deep over-parameterized neural networks, one can find an approximate stationary solution for the non-convex contrastive loss. This result is inherently different from the existing over-parameterized analysis in the supervised setting because, in contrast to learning a specific target function, unsupervised contrastive learning tries to encode the unlabeled data distribution into the neural networks, which generally has no optimal solution. Our analysis provides theoretical insights into the practical success of these unsupervised pretraining methods.
