论文标题
通过对准和统一性了解对比度表示的学习
Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
论文作者
论文摘要
对比表示学习在实践中取得了巨大的成功。在这项工作中，我们确定了与对比度损失相关的两个关键特性：（1）正面对的特征的比对（紧密），以及（2）诱导的（归一化）特征在Hypersphere上诱导的分布的均匀性。我们证明，渐近地证明，对比损失优化了这些特性，并分析了它们对下游任务的积极影响。从经验上讲，我们引入了一个可优化的度量标准来量化每个属性。关于标准视觉和语言数据集的广泛实验证实了指标和下游任务性能之间的强烈一致性。值得注意的是，对这两个指标的直接优化会导致在下游任务中具有可比或更好的性能的表示，而不是对比度学习。 项目页面：https：//tongzhouwang.info/hypersphere 代码：https：//github.com/ssnl/align_uniform，https://github.com/ssnl/moco_align_uniform
Contrastive representation learning has been outstandingly successful in practice. In this work, we identify two key properties related to the contrastive loss: (1) alignment (closeness) of features from positive pairs, and (2) uniformity of the induced distribution of the (normalized) features on the hypersphere. We prove that, asymptotically, the contrastive loss optimizes these properties, and analyze their positive effects on downstream tasks. Empirically, we introduce an optimizable metric to quantify each property. Extensive experiments on standard vision and language datasets confirm the strong agreement between both metrics and downstream task performance. Remarkably, directly optimizing for these two metrics leads to representations with comparable or better performance at downstream tasks than contrastive learning. Project Page: https://tongzhouwang.info/hypersphere Code: https://github.com/SsnL/align_uniform , https://github.com/SsnL/moco_align_uniform
