ELMo

ELMo (Embeddings from Language Models), published by the Allen Institute for AI at NAACL 2018 (arXiv:1802.05365, a best-paper award), ended the era of static word vectors. Instead of one fixed embedding per word, ELMo derived representations from a deep bidirectional language model, so 'bank' gets a different vector in 'river bank' versus 'savings bank'. Downstream models learned a task-specific linear combination of all biLM layers — lower layers capturing syntax, higher layers capturing semantics.

The results were dramatic: state of the art on six NLP tasks at once, with up to 20% relative error reduction, and ELMo embeddings could be dropped into existing models as a replacement embedding layer. It sits squarely on the bridge from word2vec to the modern era — ELMo (Feb 2018) to GPT (June 2018) to BERT (Oct 2018) — and its pretrained biLM remains available for research.