AudioLDM

AudioLDM (arXiv 2301.12503, January 2023; ICML 2023), from the University of Surrey's CVSSP group, brought latent diffusion to text-to-audio generation. It learns continuous audio representations from CLAP embeddings in a compressed latent space, training the diffusion model on audio embeddings while conditioning on text embeddings at sampling time.

Because it only needs audio data (not scarce audio-text pairs) for generative training, it is unusually compute-efficient: trained on AudioCaps with a single GPU, it reached state-of-the-art text-to-audio quality on objective and subjective metrics. It was also the first TTA system to demonstrate zero-shot text-guided audio manipulations like style transfer and inpainting. Code and checkpoints are open source.