论文标题
使用紧密耦合的老师提取的密集表示以进行排名
Distilling Dense Representations for Ranking using Tightly-Coupled Teachers
论文作者
论文摘要
我们提出了一种使用密集表示的排名方法，该方法可应用知识蒸馏以改善最近提出的后期交流COLBERT模型。具体而言，我们将知识从Colbert的表达性MaxSim运算符中提炼，以将相关得分计算为简单的点产品，从而实现了单步搜索。我们的关键见解是，在蒸馏过程中，教师模型和学生模型之间的紧密耦合可以实现更灵活的蒸馏策略，并产生更好的学费。我们从经验上表明，我们的方法改善了查询延迟，并大大减少了科尔伯特的繁重存储要求，同时仅在有效性方面做出了适度的牺牲。通过将我们的致密表示与文档扩展得出的稀疏表示形式相结合，我们能够使用BERT使用BERT（较慢的数量级）来接近标准的交叉编码重新编码器的有效性。
We present an approach to ranking with dense representations that applies knowledge distillation to improve the recently proposed late-interaction ColBERT model. Specifically, we distill the knowledge from ColBERT's expressive MaxSim operator for computing relevance scores into a simple dot product, thus enabling single-step ANN search. Our key insight is that during distillation, tight coupling between the teacher model and the student model enables more flexible distillation strategies and yields better learned representations. We empirically show that our approach improves query latency and greatly reduces the onerous storage requirements of ColBERT, while only making modest sacrifices in terms of effectiveness. By combining our dense representations with sparse representations derived from document expansion, we are able to approach the effectiveness of a standard cross-encoder reranker using BERT that is orders of magnitude slower.
