autoevals
Open-source library for evaluating AI model outputs with LLM judges and classic metrics.
About
autoevals is Braintrust's open-source evaluation library, available in both TypeScript and Python, for quickly evaluating AI model outputs. It bundles LLM-as-a-judge scorers (Factuality, ClosedQA, ContextRelevancy, Faithfulness, and more), heuristic scorers like Levenshtein distance, and statistical metrics like BLEU, all sharing one Score interface and evaluation templates.
Many of the model-graded evaluations are adapted from OpenAI's evals project but reimplemented so they run on individual examples with tweakable prompts and debuggable outputs. The library works standalone without the Braintrust platform, though LLM-judge scorers naturally require API keys for the judge models they call.
What it can do
- LLM-as-a-judge scorers (factuality, faithfulness, QA)
- Heuristic scorers like Levenshtein
- Statistical metrics like BLEU
- Shared Score interface across TS and Python
- Standalone use without the Braintrust platform
Setup
npm install autoevals for TypeScript or pip install autoevals for Python, then import scorers directly.
Prerequisites
API keys for whichever judge models you use (e.g. OpenAI or Anthropic); Python 3.9+ or Node.
Use cases
Works with
Trust score — 8/10
Scored 8/10 (verified tier, metrics-v1). Evidence: Verified via vendor site/docs/GitHub and reputable coverage, wave 79; Trending score 76/100 from researcher assessment.
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.