HELM
Stanford's holistic, reproducible evaluation framework for foundation models.
About
HELM, the Holistic Evaluation of Language Models, is the open-source Python framework created by Stanford's Center for Research on Foundation Models to make model evaluation reproducible and transparent. It standardizes datasets and benchmarks in one format, runs models from many providers through a unified interface, and measures far more than accuracy: efficiency, calibration, robustness, bias, and toxicity all get first-class metrics.\n\nThe tooling includes a command line runner, a results summarizer, a web UI for inspecting individual prompts and responses, and official leaderboards spanning capabilities, safety, and vision-language models. HELM entered maintenance mode in June 2026, but its datasets, scenario definitions, and leaderboard history remain a reference implementation that many newer eval harnesses borrowed from. It is installed from PyPI as crfm-helm and ships under Apache 2.0.
What it can do
- Standardized benchmark scenarios including MMLU-Pro, GPQA, IFEval, and WildBench
- Unified interface across OpenAI, Anthropic, Google, and other providers
- Metrics beyond accuracy: efficiency, bias, toxicity, robustness
- Web UI plus public leaderboards for capabilities, safety, and vision-language
- Apache 2.0 licensed, pip-installable framework with extensive docs
Setup
Run pip install crfm-helm, then use helm-run to execute benchmark suites and helm-server to browse results.
Prerequisites
Python 3 and API keys for any hosted models you want to evaluate.
Use cases
Works with
Trust score — 8/10
Scored 8/10 (verified tier, metrics-v1). Evidence: Verified via vendor site/docs/GitHub and reputable coverage, wave 100; Trending score 80/100 from researcher assessment.
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.