OpenAI Evals
OpenAI's framework for evaluating LLMs and LLM systems, with a registry covering logic, coding, and agentic tasks.
About
OpenAI Evals is the framework OpenAI released for systematically testing language models and systems built on them: evals are defined as YAML plus Python logic, run against model samples, and scored by deterministic checks or model-graded rubrics. The public registry includes evals for logic puzzles, coding, and longer agentic workflows.
It is a solid, simple abstraction for regression-testing model behavior across releases, and the eval registry gives teams a starting set of tasks to adapt. One honest caveat: the repository's maintenance pace has slowed (last significant activity in early 2026), with much of OpenAI's eval effort moving elsewhere — still useful, but check the registry's freshness for your domain before committing.
What it can do
- YAML-defined evals with Python scoring logic
- Public registry of logic, coding, and agentic evals
- Deterministic and model-graded scoring modes
- Regression testing across model versions
- Extensible for private, domain-specific eval suites
Setup
Clone openai/evals, pip install the requirements, set OPENAI_API_KEY, then run oaieval <model> <eval-name> to execute a registry eval.
Prerequisites
Python 3.9+; OpenAI API access for model-graded evals.
Use cases
Works with
Trust score — 7/10
Scored 7/10 (metrics tier, metrics-v1). Evidence: 19,607 GitHub stars (openai/evals); last pushed 2026-04-14 (maintenance pace slowed - flagged in entry); license not asserted in GitHub API response; OpenAI-authored; officialUrl https://github.com/openai/evals curl-verified 200. Manual review: YAML+Python eval definition and registry model verified against repo README; freshness caveat noted honestly. License: check repo.
★ 19,607 GitHub stars
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.