论文标题
在非常低的资源语言建模中上下文的重要性
The Importance of Context in Very Low Resource Language Modeling
论文作者
论文摘要
本文研究了少于1万个句子时，研究了非常低的资源语言模型。我们发现，在非常低的资源方案中，统计N-Gram语言模型的表现优于最先进的神经模型。我们的实验表明，这主要是由于前者的重点放在当地环境上。因此，我们介绍了三种方法，以提高神经模型在低资源环境中的性能，发现限制模型的自我注意力是最有效的一种方法，可以改善下游任务，例如NLI，POS标签，并在我们测试的语言中提高了5％的标签：英语，印地语和土耳其语。
This paper investigates very low resource language model pretraining, when less than 100 thousand sentences are available. We find that, in very low resource scenarios, statistical n-gram language models outperform state-of-the-art neural models. Our experiments show that this is mainly due to the focus of the former on a local context. As such, we introduce three methods to improve a neural model's performance in the low-resource setting, finding that limiting the model's self-attention is the most effective one, improving on downstream tasks such as NLI and POS tagging by up to 5% for the languages we test on: English, Hindi, and Turkish.
