Hallo3

Hallo3, from Fudan University's Generative Vision lab, is the first portrait-image animation method built on a pretrained transformer-based video backbone rather than U-Net diffusion. A causal 3D VAE plus stacked transformer layers forms an identity reference network that keeps faces consistent across sequences, while speech-audio conditioning and motion frames drive continuous video — handling non-frontal poses, dynamic foregrounds, and immersive backgrounds better than prior methods.

Code and a 120-hour training dataset are public (GitHub fudan-generative-vision/hallo3, arXiv 2412.00733). Limitations: it was tested on H100s, so consumer-GPU inference needs patience or quantization, and as a research release there is no hosted API or product wrapper — you run the code yourself.