论文标题
HPO：我们不会再上当了
HPO: We won't get fooled again
论文作者
论文摘要
高参数优化（HPO）是一个良好的研究领域。但是，HPO管道中组件的效果和相互作用尚未得到很好的研究。然后，我们问自己：HPO的景观是否会被用于评估各个配置的管道偏见吗？为了解决这个问题，我们建议使用健身景观分析分析HPO管道对HPO问题的影响。特别是，我们研究了DS-2019 HPO基准数据集，寻找可能表明评估管道故障的模式，并将其与HPO性能联系起来。我们的主要发现是：（i）在大多数情况下，大量不同的超参数（即多种配置）产生相同的不良表现，很可能与多数类预测模型有关； （ii）在这些情况下，观察到观察到的健康和平均健身之间的关系恶化，可能会使基于本地搜索的HPO策略的部署更加困难。最后，我们得出的结论是，HPO管道定义可能会对HPO景观产生负面影响。
Hyperparameter optimization (HPO) is a well-studied research field. However, the effects and interactions of the components in an HPO pipeline are not yet well investigated. Then, we ask ourselves: can the landscape of HPO be biased by the pipeline used to evaluate individual configurations? To address this question, we proposed to analyze the effect of the HPO pipeline on HPO problems using fitness landscape analysis. Particularly, we studied the DS-2019 HPO benchmark data set, looking for patterns that could indicate evaluation pipeline malfunction, and relate them to HPO performance. Our main findings are: (i) In most instances, large groups of diverse hyperparameters (i.e., multiple configurations) yield the same ill performance, most likely associated with majority class prediction models; (ii) in these cases, a worsened correlation between the observed fitness and average fitness in the neighborhood is observed, potentially making harder the deployment of local-search based HPO strategies. Finally, we concluded that the HPO pipeline definition might negatively affect the HPO landscape.
