论文标题
干草堆中的针头：对MTURK上的高级工人进行分析以进行摘要
Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarization
论文作者
论文摘要
为了防止在低质量注释中资源的昂贵和效率低下的利用，我们希望一种方法来创建可靠的注释者池，这些注释者可以有效地完成困难的任务，例如评估自动汇总。因此，我们通过两步管道研究了高质量的亚马逊机械土耳其人工人的招募。我们表明，我们可以在进行评估并获得具有相似限制资源的高级注释之前成功地过滤掉不优秀的工人。尽管我们的工人之间表现出了强烈的共识和cloudresearch的工作者，但他们与数据子集的专家判断的一致性并不预期，并且需要进一步的正确性培训。本文仍然是在其他具有挑战性的注释任务中招募合格注释者的最佳实践。
To prevent the costly and inefficient use of resources on low-quality annotations, we want a method for creating a pool of dependable annotators who can effectively complete difficult tasks, such as evaluating automatic summarization. Thus, we investigate the recruitment of high-quality Amazon Mechanical Turk workers via a two-step pipeline. We show that we can successfully filter out subpar workers before they carry out the evaluations and obtain high-agreement annotations with similar constraints on resources. Although our workers demonstrate a strong consensus among themselves and CloudResearch workers, their alignment with expert judgments on a subset of the data is not as expected and needs further training in correctness. This paper still serves as a best practice for the recruitment of qualified annotators in other challenging annotation tasks.
