Meta is trying to bridge the communication gap between languages that exists in our multilingual world. They have launched SeamlessM4T, which is a foundational multilingual and multitask model that can translate between languages across speech and text.
“The world we live in has never been more interconnected—the global proliferation of the internet, mobile devices, social media, and communication platforms gives people access to more multilingual content than ever before. In such a context, having an on-demand ability to communicate and understand information in any language becomes increasingly important. While such a capability has long been dreamed of in science fiction, AI is on the verge of bringing this vision into technical reality,” Meta wrote in a blog post.
SeamlessM4T already supports automatic speech recognition, speech-to-text translation, and text-to-text translation for almost 100 languages. It also can do speech-to-speech and text-to-speech translation for almost 100 input languages and 35 output languages.
The project has been released under the CC BY-NC 4.0 license in order to allow researchers to build on it.
Along with releasing SeamlessM4T, Meta is also releasing SeamlessAlign, which is a dataset for multimodal translation that includes 270,000 hours of speech and text alignments.
According to Meta, existing speech-to-speech and speech-to-text programs only cover a fraction of the world’s languages and this project represents a breakthrough in the number of languages covered.
It builds on Meta’s existing work in this field, including No Language Left Behind, Universal Speech Translator, SpeechMatrix, and Massively Multilingual Speech.
Meta also described the steps it took to build the model responsibly. The company followed its five pillars of Responsible AI, and conducted toxicity and bias research to understand areas of the mode that could be sensitive. It is also beginning to conduct gender bias evaluations on the model.
“Our work around safety and security is an ongoing effort. We’ll continue to research and take action in this area to continuously improve SeamlessM4T and reduce any instances of toxicity we see in the model,” Meta wrote.
