RoBERTa

RoBERTa, published by Meta (then Facebook AI) with University of Washington collaborators in July 2019, is a replication study that became a landmark. The authors showed BERT was significantly undertrained and that careful hyperparameter and data choices alone could match or exceed every model published after BERT. Key changes: training longer on much more data (160GB including CC-News, OpenWebText, and Stories), removing the next-sentence-prediction objective, training on full-length sequences, and dynamically changing the masking pattern.

The resulting model set state of the art on GLUE, RACE, and SQuAD at the time, and its training recipe became the default for encoder pretraining for years. Models and code were released openly. A historical entry: essential context for understanding the 2019-2021 NLP era, though generative models have since eclipsed encoders for most applications.