TinyLlama

TinyLlama is a 1.1-billion-parameter open-source language model from SUTD's Lu Wei team, released in early 2024. It follows the Llama 2 architecture and tokenizer exactly, but was trained on roughly 3 trillion tokens in about four months on just 16 A100 GPUs, an early demonstration that small models trained far beyond Chinchilla-optimal token counts punch above their weight.

It outperformed comparable-size open models like OPT-1.3B and Pythia-1.4B on downstream tasks while fitting in about 550MB of RAM, making it a staple for mobile deployment experiments and a drop-in platform for the Llama ecosystem. The full pretraining was done in the open with intermediate checkpoints published.