论文标题
异构校准：事后模型不合时宜的框架，用于改进概括
Heterogeneous Calibration: A post-hoc model-agnostic framework for improved generalization
论文作者
论文摘要
我们介绍了异质校准的概念，该校准应用于事后模型不合时宜的转换，以改善在二进制分类任务上的AUC性能的模型输出。我们考虑过度自信的模型，其在训练数据与测试数据上的性能明显更好，并给出了为什么它们可能不足以利用数据中适度有效的简单模式。我们将这些简单模式称为特征空间的异质分区，并从理论上显示完美校准每个分区可以分别优化AUC。这给出了异质校准的一般范式作为事后程序，通过基于树的算法，通过基于树的算法确定特征空间的异质分区，并将事后校准技术应用于每个分区以改善AUC。尽管此框架的理论最优性适用于任何模型，但我们专注于深神经网络（DNN），并在各种开源数据集上测试该范式的最简单实例化。实验证明了该框架的有效性以及应用较高表现分配方案以及更有效的校准技术的未来潜力。
We introduce the notion of heterogeneous calibration that applies a post-hoc model-agnostic transformation to model outputs for improving AUC performance on binary classification tasks. We consider overconfident models, whose performance is significantly better on training vs test data and give intuition onto why they might under-utilize moderately effective simple patterns in the data. We refer to these simple patterns as heterogeneous partitions of the feature space and show theoretically that perfectly calibrating each partition separately optimizes AUC. This gives a general paradigm of heterogeneous calibration as a post-hoc procedure by which heterogeneous partitions of the feature space are identified through tree-based algorithms and post-hoc calibration techniques are applied to each partition to improve AUC. While the theoretical optimality of this framework holds for any model, we focus on deep neural networks (DNNs) and test the simplest instantiation of this paradigm on a variety of open-source datasets. Experiments demonstrate the effectiveness of this framework and the future potential for applying higher-performing partitioning schemes along with more effective calibration techniques.
