论文标题
监视机器学习系统的亚组性能的有效框架
An Efficient Framework for Monitoring Subgroup Performance of Machine Learning Systems
论文作者
论文摘要
监视机器学习系统部署后对于确保系统的可靠性至关重要。特别重要的是监视所有数据亚组中机器学习系统的性能（亚群）的问题。实际上，此过程可能会非常昂贵，因为数据子组的数量随输入功能的数量呈指数增长，并且标记数据以评估每个亚组的性能的过程成本很高。在本文中，我们提出了一个有效的框架，用于监视机器学习系统的亚组性能。具体而言，我们旨在使用有限数量的标记数据找到具有最差性能的数据子组。我们在数学上将此问题提出为昂贵的黑盒目标功能的优化问题，然后建议使用贝叶斯优化解决此问题。我们对各种现实世界数据集和机器学习系统的实验结果表明，我们提出的框架可以有效，有效地检索最差的数据亚组。
Monitoring machine learning systems post deployment is critical to ensure the reliability of the systems. Particularly importance is the problem of monitoring the performance of machine learning systems across all the data subgroups (subpopulations). In practice, this process could be prohibitively expensive as the number of data subgroups grows exponentially with the number of input features, and the process of labelling data to evaluate each subgroup's performance is costly. In this paper, we propose an efficient framework for monitoring subgroup performance of machine learning systems. Specifically, we aim to find the data subgroup with the worst performance using a limited number of labeled data. We mathematically formulate this problem as an optimization problem with an expensive black-box objective function, and then suggest to use Bayesian optimization to solve this problem. Our experimental results on various real-world datasets and machine learning systems show that our proposed framework can retrieve the worst-performing data subgroup effectively and efficiently.
