Longformer

Longformer, published by the Allen Institute for AI in April 2020 (arXiv:2004.05150), attacked the transformer's core scaling problem: full self-attention grows quadratically with sequence length, capping BERT-era models at 512 tokens. Its answer was a drop-in attention replacement combining sliding-window local attention with a small set of task-specific global tokens, bringing complexity down to linear and letting a single model read documents of 4,096 tokens and beyond.

The pretrained longformer-base-4096, initialized from RoBERTa and further trained on long documents, set new state-of-the-art results on WikiHop and TriviaQA and became the standard long-document baseline for years. It directly influenced the efficient-attention wave that followed — BigBird, sparse transformers, and eventually the long-context arms race of the LLM era. The code and checkpoints remain open via the AllenAI organization.