ELECTRA

ELECTRA, from Stanford and Google Brain (ICLR 2020), replaced BERT's masked-language-modeling objective with replaced token detection: a small generator network corrupts the input with plausible alternative tokens, and the discriminator (the model you actually use) learns to detect which tokens were replaced. Because the loss applies to every input token rather than just the 15% that are masked, pretraining is far more compute-efficient.

The efficiency gains were dramatic at small scale: an ELECTRA model trained on a single GPU for four days outperformed GPT on the GLUE benchmark despite GPT using 30x more compute, and at large scale it matched RoBERTa and XLNet with less than a quarter of their compute. Google released official code and checkpoints. A historical landmark entry in efficient pretraining, though the field has since moved to generative pretraining at scale.