SeamlessM4T

SeamlessM4T is Meta's massively multilingual, multimodal translation model: a single system that handles speech-to-speech, speech-to-text, text-to-speech, text-to-text, and ASR instead of chaining separate models. Version 2 ships a 2.3B large checkpoint built on a w2v-BERT 2.0 speech encoder, an NLLB-style text decoder, and a unit-based HiFi-GAN vocoder, trained over 470,000 hours of automatically aligned speech translations. The SeamlessExpressive extension preserves the speaker's voice and prosody across languages, and SeamlessStreaming targets real-time use. Weights are CC-BY-NC 4.0, so it is a research and non-commercial asset that needs legal review before product use. Translation quality tracks resource levels per language, and the large model wants a serious GPU (about 8.6 GB on disk, ~4.3 GB VRAM in fp16).

Team-reported: state-of-the-art results on Fleurs and CVSS speech translation benchmarks (papers arXiv 2308.11596, 2312.05187)