GPT-J

GPT-J-6B, released by EleutherAI on June 9, 2021, was the first open model to match GPT-3-class performance at the 6B scale. A 6-billion-parameter autoregressive decoder-only transformer with 28 layers and a 2048-token context window, it was trained on The Pile using the Mesh Transformer JAX library on Google TPU Research Cloud compute.

At release it performed comparably to GPT-3 6.7B (Curie) on downstream tasks, and its Apache 2.0 weights made it the workhorse of open LLM experimentation in 2021-2022, spawning a large fine-tune ecosystem. Development stopped in 2021 and the official web demo is long offline, but the weights remain widely mirrored. A historical landmark entry: it proved a volunteer collective could replicate frontier-lab results.