RETRO

RETRO, short for Retrieval-Enhanced Transformer, is DeepMind's December 2021 answer to the brute-force scaling of language models. Instead of memorizing the world in parameters, RETRO splits its input into chunks and retrieves similar text passages from a 2-trillion-token database, folding them in through chunked cross-attention. A frozen BERT retriever finds the passages, so no retriever network needs training.

The result was striking for its time: a 7.5-billion-parameter RETRO matched GPT-3 and Jurassic-1 on the Pile while using 25 times fewer parameters, and the architecture could be improved after training simply by growing the retrieval database. It also made the model's behavior more inspectable, since researchers could see which passages informed a prediction. RETRO opened the research path toward semi-parametric language models and influenced later retrieval-augmented generation work. DeepMind never released RETRO's weights publicly, so openWeights is false; it remains a research landmark rather than a usable model.