论文标题
差异信息神经估计
Diffeomorphic Information Neural Estimation
论文作者
论文摘要
相互信息（MI）和条件共同信息（CMI）是信息理论中的多功能工具，能够自然地衡量随机变量之间的统计依赖性，因此它们通常在几个统计和机器学习任务中具有核心兴趣，例如有条件的独立性测试和表示学习。但是，由于棘手的配方，估计CMI甚至MI是臭名昭著的。在这项研究中，我们介绍了用餐（差异信息神经估计量） - 一种新的方法，用于估计连续随机变量的CMI，灵感来自CMI在DIFFEMERGERIC图上的不变性。我们表明，可以用遵循更简单的分布的适当替代物代替感兴趣的变量，从而可以通过分析解决方案进行有效评估CMI。此外，我们在三个重要任务中与最新任务相比，我们证明了所提出的估计器的质量，包括估计MI，CMI及其在条件独立性测试中的应用。经验评估表明，用餐在所有任务中始终优于竞争对手，并且能够很好地适应复杂和高维的关系。
Mutual Information (MI) and Conditional Mutual Information (CMI) are multi-purpose tools from information theory that are able to naturally measure the statistical dependencies between random variables, thus they are usually of central interest in several statistical and machine learning tasks, such as conditional independence testing and representation learning. However, estimating CMI, or even MI, is infamously challenging due the intractable formulation. In this study, we introduce DINE (Diffeomorphic Information Neural Estimator)-a novel approach for estimating CMI of continuous random variables, inspired by the invariance of CMI over diffeomorphic maps. We show that the variables of interest can be replaced with appropriate surrogates that follow simpler distributions, allowing the CMI to be efficiently evaluated via analytical solutions. Additionally, we demonstrate the quality of the proposed estimator in comparison with state-of-the-arts in three important tasks, including estimating MI, CMI, as well as its application in conditional independence testing. The empirical evaluations show that DINE consistently outperforms competitors in all tasks and is able to adapt very well to complex and high-dimensional relationships.
