论文标题
不要信任模型，因为它有信心：在基于在线学习中的学生成功预测因素，揭示和表征未知的未知数
Do Not Trust a Model Because It is Confident: Uncovering and Characterizing Unknown Unknowns to Student Success Predictors in Online-Based Learning
论文作者
论文摘要
学生成功模型可能很容易发展弱点，即，由于模型创建过程中的表示不足，因此难以准确分类的示例。这种弱点是破坏用户信任的主要因素之一，因为例如，模型预测可能会导致讲师不干预有需要的学生。在本文中，我们揭露了检测和表征学生成功预测未知未知数的需求，以便更好地了解模型何时可能失败。未知的未知数包括该模型对其预测高度自信的学生，但实际上是错误的。因此，在评估预测质量时，我们不能仅仅依靠模型的信心。我们首先引入了一个框架，以识别和表征未知的未知数。然后，我们使用定量分析和对教师的访谈评估了其对从翻转课程和在线课程收集的日志数据的信息。我们的结果表明，未知的未知数是该领域的关键问题，我们的框架可以用于支持其检测。源代码可在https://github.com/epfl-ml4ed/unknown-inknowns上找到。
Student success models might be prone to develop weak spots, i.e., examples hard to accurately classify due to insufficient representation during model creation. This weakness is one of the main factors undermining users' trust, since model predictions could for instance lead an instructor to not intervene on a student in need. In this paper, we unveil the need of detecting and characterizing unknown unknowns in student success prediction in order to better understand when models may fail. Unknown unknowns include the students for which the model is highly confident in its predictions, but is actually wrong. Therefore, we cannot solely rely on the model's confidence when evaluating the predictions quality. We first introduce a framework for the identification and characterization of unknown unknowns. We then assess its informativeness on log data collected from flipped courses and online courses using quantitative analyses and interviews with instructors. Our results show that unknown unknowns are a critical issue in this domain and that our framework can be applied to support their detection. The source code is available at https://github.com/epfl-ml4ed/unknown-unknowns.
