论文标题
T模型：零射击的跨模式转换的翻译模块翻译
T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translation
论文作者
论文摘要
我们提出了一种新的方法，可以在语音和文本之间执行零射击的交叉模式转移，以进行翻译任务。多语言语音和文本在联合固定大小的表示空间中编码。然后，我们比较不同的方法来解码这些多模式和多语言固定尺寸表示形式，从而使语言和模态之间的零弹性翻译。我们所有的模型均经过训练，而无需跨模式标记的翻译数据。尽管有固定尺寸的表示，但我们在几个文本和语音翻译任务上取得了非常具竞争力的结果。特别是，我们大大改善了在必须使用的零击语音翻译的最先进。将语音解码器纳入我们的框架中，我们介绍了零拍的直接语音到语音和文本到语音翻译的第一个结果。
We present a new approach to perform zero-shot cross-modal transfer between speech and text for translation tasks. Multilingual speech and text are encoded in a joint fixed-size representation space. Then, we compare different approaches to decode these multimodal and multilingual fixed-size representations, enabling zero-shot translation between languages and modalities. All our models are trained without the need of cross-modal labeled translation data. Despite a fixed-size representation, we achieve very competitive results on several text and speech translation tasks. In particular, we significantly improve the state-of-the-art for zero-shot speech translation on Must-C. Incorporating a speech decoder in our framework, we introduce the first results for zero-shot direct speech-to-speech and text-to-speech translation.
