论文标题
上下文单词表示的多语言对齐
Multilingual Alignment of Contextual Word Representations
论文作者
论文摘要
我们提出了评估和加强上下文嵌入对齐的程序，并表明它们在分析和改善多语言BERT方面有用。特别是，在我们提出的对齐程序之后，与基本模型相比，伯特在XNLI上表现出显着提高的零击性能，与保加利亚人和希腊语相比，伪造的伪用户训练训练模型非常匹配。此外，为了衡量对齐程度，我们介绍了单词检索的上下文版本，并表明它与下游零弹性传输良好相关。使用此词检索任务，我们还分析了BERT并发现它表现出系统性缺陷，例如对于用不同脚本编写的开放类词性部分和单词对的比对更糟糕的对齐，这些脚本通过对齐过程进行了纠正。这些结果支持上下文对齐，作为理解大型多语言预训练模型的有用概念。
We propose procedures for evaluating and strengthening contextual embedding alignment and show that they are useful in analyzing and improving multilingual BERT. In particular, after our proposed alignment procedure, BERT exhibits significantly improved zero-shot performance on XNLI compared to the base model, remarkably matching pseudo-fully-supervised translate-train models for Bulgarian and Greek. Further, to measure the degree of alignment, we introduce a contextual version of word retrieval and show that it correlates well with downstream zero-shot transfer. Using this word retrieval task, we also analyze BERT and find that it exhibits systematic deficiencies, e.g. worse alignment for open-class parts-of-speech and word pairs written in different scripts, that are corrected by the alignment procedure. These results support contextual alignment as a useful concept for understanding large multilingual pre-trained models.
