What is an LLM

The definition

A large language model is a neural network, almost always a transformer, trained on vast amounts of text to predict what comes next. Given a sequence of tokens, it outputs a probability distribution over the vocabulary for the following token. That is the entire core capability. Everything else, answering questions, writing code, translating, summarizing, is this prediction step applied repeatedly, with the model's own output fed back in as new input.

The "large" refers to three kinds of scale at once. The models have huge numbers of parameters: GPT-3, released in 2020, had 175 billion. They train on huge datasets: trillions of tokens drawn from web pages, books, and code. And they consume huge amounts of compute: weeks or months on thousands of specialized chips. Remove any one of the three and the results degrade. The field's central empirical finding, formalized as scaling laws around 2020, is that performance improves predictably as you scale all three together.

How one gets built

Building a frontier language model happens in stages.

Pretraining comes first. The model reads through its enormous text corpus, learning to predict each next token. Nobody labels anything; the text itself is the supervision, since every token is preceded by its context. This stage teaches the model grammar, facts, reasoning patterns, coding idioms, and every bias present in the data. It is by far the most expensive stage and produces a base model: a fluent text predictor with no particular interest in being helpful.

Supervised fine-tuning comes next. The base model trains further on curated examples of instructions paired with good responses: questions with answers, tasks with demonstrations. This is a small fraction of the pretraining data but high in quality. It teaches the model the format of being an assistant rather than a text completer.

Then reinforcement learning from human feedback, RLHF. Human annotators rank several of the model's responses to the same prompt, and a reward model learns to predict those rankings. The language model is then optimized to score well according to the reward model. This stage, introduced at scale with InstructGPT in 2022, is what made models follow instructions reliably and avoid the worst outputs. It is also where much of a model's "personality" and refusal behavior gets shaped.

The 2020 surprise

GPT-3's 2020 release changed what people believed these models could do. Earlier language models needed fine-tuning on task-specific data to be useful for any particular job. GPT-3 could perform tasks it was never explicitly trained for, just from examples placed in the prompt. Show it a few English-to-French pairs and it translates. Show it a few question-answer pairs and it answers. This in-context learning, sometimes called few-shot learning, was not programmed in. It emerged from scale.

The public breakthrough came in November 2022, when ChatGPT put a fine-tuned descendant of GPT-3 behind a simple chat interface. Within months it was the fastest-growing consumer product in history. GPT-4 followed in March 2023 with stronger reasoning and multimodal input. The base technology in all of them was the transformer from 2017, scaled up and trained with the three-stage recipe.

What the model knows, and what it does not

It helps to be precise about what "knowledge" means here. The model's weights encode statistical patterns from its training data: which words follow which, which facts co-occur, how arguments are structured. When it answers a factual question correctly, it is reproducing a pattern that was frequent and consistent in training. It has no database of facts, no mechanism for checking truth, and no awareness of when its training data was wrong.

This is why the same model can write a flawless explanation of quantum tunneling and confidently invent a paper that does not exist. Both outputs are produced by the same process: generating the most plausible continuation. Plausibility and truth correlate well on topics the training data covered thoroughly, and poorly everywhere else. The article on hallucinations covers this failure mode in detail.

The model's knowledge is also frozen at training time. It knows nothing about events after its training data was collected, and it cannot learn from conversation. Each session starts from the same weights. Techniques like retrieval-augmented generation work around this by fetching fresh information at query time, but the model itself does not update.

Open and closed

Language models come in two broad distributions. Proprietary models, served through APIs, keep their weights private; you send text in and get text out. Open-weight models publish the trained parameters, letting anyone download and run them, fine-tune them, and inspect them. The open models lag the frontier somewhat but have improved rapidly, and they matter because they let researchers study the technology and let builders deploy without depending on a vendor.

The cost structure follows the training-inference split. Training a frontier model is a nine-figure undertaking. Running inference on one costs fractions of a cent per request. This gap is why a handful of organizations train the largest models while thousands of products are built on top of them.

What "understanding" means here

Whether language models understand anything is a live philosophical debate and not one this article will settle. The practical view: they model the statistical structure of language at a depth that captures a great deal of the structure of thought, because thought expressed in language leaves statistical traces. That makes them extraordinarily useful for tasks that look like "produce the text a competent person would produce here," and unreliable for tasks that require verifying claims against reality. Knowing which kind of task you are asking for is the most important skill in using them well.