论文标题
TREC 2019公平排名轨道概述
Overview of the TREC 2019 Fair Ranking Track
论文作者
论文摘要
TREC Fair排名轨道的目标是开发一个基准，除了经典的相关性概念外，还要对不同内容提供商进行公平性评估检索系统。作为基准的一部分，我们通过评估协议定义了标准化的公平指标，并为公平排名问题发布了数据集。 2019年的任务重点是重新撰写学术论文摘要。目的是公平地代表来自系统提交时间的几个小组的相关作者。因此，该轨道强调了在各种组定义中具有强大性能的系统的开发。向参与者提供了来自语义学者的QueryLog数据（查询，文档和相关性）。本文概述了该曲目的概述，包括任务定义，数据描述和注释过程以及提交系统性能的比较。
The goal of the TREC Fair Ranking track was to develop a benchmark for evaluating retrieval systems in terms of fairness to different content providers in addition to classic notions of relevance. As part of the benchmark, we defined standardized fairness metrics with evaluation protocols and released a dataset for the fair ranking problem. The 2019 task focused on reranking academic paper abstracts given a query. The objective was to fairly represent relevant authors from several groups that were unknown at the system submission time. Thus, the track emphasized the development of systems which have robust performance across a variety of group definitions. Participants were provided with querylog data (queries, documents, and relevance) from Semantic Scholar. This paper presents an overview of the track, including the task definition, descriptions of the data and the annotation process, as well as a comparison of the performance of submitted systems.
