Inspect AI
Open-source framework for frontier AI evaluations, from the UK AI Security Institute.
About
Inspect is the evaluation framework built by the UK AI Security Institute (renamed from the AI Safety Institute in 2025). It is the tooling the Institute itself uses for pre-deployment evaluations of frontier models, which gives it unusual credibility: this is not a demo harness, it is the instrument behind published government safety assessments of models from Anthropic and others.
The framework models an evaluation as Tasks composed of Datasets, Solvers (the thing being tested, which can use tools), and Scorers, with sandboxed execution via Docker so agent evals can run code safely. It supports model-graded scoring, multi-step agentic evaluations, and parallel runs, and ships with an Inspect View web UI for digging through results. The companion inspect_evals repository collects 100+ real evaluations written by the Institute and collaborators. It is MIT licensed and installs with pip install inspect-ai.
What it can do
- Task/Dataset/Solver/Scorer evaluation model
- Sandboxed tool execution via Docker
- Model-graded and custom scorers
- Agentic multi-step evaluations with tool use
- Inspect View web UI for result analysis
- 100+ ready-made evals in the inspect_evals companion repo
Setup
- Run
pip install inspect-ai. - Set your model provider API key.
- Write a Task with a dataset, solver, and scorer (or use one from inspect_evals).
- Run with
inspect evaland review results in Inspect View.
Prerequisites
Python 3.10+, API access to the model under test; Docker for sandboxed evals.
Use cases
Works with
Trust score — 8/10
Repo UKGovernmentBEIS/inspect_ai verified (3.0k stars), MIT license confirmed via GitHub API, docs at inspect.aisi.org.uk return 200. Framework architecture and government pre-deployment use confirmed via multiple secondary sources. No official product logo found, so logoUrl is null.
★ 3,000 GitHub stars
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.