DeepEval
Pytest-native LLM evaluation: unit-test your LLM outputs in CI.
About
DeepEval is an open-source LLM evaluation framework that treats evals like software tests: write pytest-style test cases asserting on LLM outputs, score them with 14+ built-in metrics (answer relevancy, faithfulness, hallucination, toxicity, RAG metrics, agentic metrics), and run them in CI. Metrics can use LLM judges, deterministic checks, or custom scorers. Backed by Confident AI's commercial platform for hosted eval management, but the framework itself is Apache-2.0 and runs locally. 18.8k stars, 2.6M PyPI downloads/month.
What it can do
- **Pytest-native**: evals as unit tests, runnable in CI pipelines
- **14+ metrics**: relevancy, faithfulness, hallucination, toxicity, bias, RAG and agentic metrics
- **Custom metrics**: bring your own scorers and judges
- **Red teaming**: adversarial testing for vulnerabilities
- **Confident AI integration**: hosted dashboards, regression tracking (optional)
Setup
```bash
pip install deepeval
```
```python
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric
test_case = LLMTestCase(input="What is 2+2?", actual_output="4")
assert_test(test_case, [AnswerRelevancyMetric(threshold=0.7)])
```
Set an LLM API key (e.g. OPENAI_API_KEY) for judge-based metrics. Docs: deepeval.com/docs.
Prerequisites
Python 3.9+. An LLM API key for judge-based metrics (deterministic metrics work offline).
Use cases
Works with
Trust score — 8/10
Scored 8/10 (verified tier, metrics-v1). Evidence: 18,773 GitHub stars (log-scaled vs catalog peers); blended with ~2,584,198 monthly package downloads; last push 0d ago; 11 releases/12mo; 720 open issues. Manual review: pytest-native LLM evals, 14+ metrics, 18.8k stars; judge metrics inherit judge bias — calibrate, don't blindly trust; commercial platform is the monetization path. License: Apache-2.0.
★ 18,773 GitHub stars
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 8.4/10
- Maintenance · recency, releases, issue load
- 7.9/10
- Openness · license + docs
- 10.0/10
- Manual review · sanity read + red-flag check
- 8.0/10
- · 18,773 GitHub stars (log-scaled vs catalog peers)
- · blended with ~2,584,198 monthly package downloads
- · last push 0d ago; 11 releases/12mo; 720 open issues
- · manual review: pytest-native LLM evals, 14+ metrics, 18.8k stars; judge metrics inherit judge bias — calibrate, don't blindly trust; commercial platform is the monetization path
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.