DistilBERT

DistilBERT showed that knowledge distillation could compress a large pretrained transformer without major accuracy loss. Hugging Face's team trained a 6-layer, 66-million-parameter student against BERT-base using a triple loss combining masked language modeling, soft-target distillation, and cosine embedding alignment, initializing the student from every other BERT layer.

Released in October 2019 (arXiv 1910.01108, NeurIPS workshop), it runs 60% faster at inference while retaining 97% of BERT's GLUE score, and became one of the most downloaded models on the Hugging Face Hub, a template for the whole model-compression wave. Checkpoints are Apache 2.0 and small enough at around 250MB to run on mobile and edge devices without quantization.