InstructGPT

InstructGPT is OpenAI's fine-tuned version of GPT-3, announced in January 2022 with the full paper 'Training Language Models to Follow Instructions with Human Feedback' posted to arXiv that March. It introduced the three-stage RLHF recipe that now underpins nearly every assistant model: supervised fine-tuning on human-written demonstrations, training a reward model on human rankings of outputs, then optimizing the policy with reinforcement learning (PPO).

The paper's most cited result: human raters preferred outputs from the 1.3B InstructGPT model over the 175B base GPT-3 — a hundred times larger — on instruction-following tasks. Alignment technique, it turned out, could substitute for raw scale, while also reducing toxic and untruthful outputs. Three sizes were trained: 1.3B, 6B, and 175B.

InstructGPT never shipped as a public product under its own name — it was served through the OpenAI API and its techniques flowed directly into ChatGPT later that year. The weights are proprietary and closed; its importance is methodological, as the blueprint for the entire modern assistant era.