Promptfoo
Developer-first LLM evals and red teaming — now part of OpenAI.
About
Promptfoo is a CLI and library for evaluating and red-teaming LLM apps: define evals in YAML, score prompts and models side-by-side, scan for vulnerabilities (jailbreaks, prompt injection, PII leaks), and gate it all in CI. Evals run 100% locally — your prompts never leave your machine — and it works with any provider or language. 25.9k stars, MIT, 3.5M npm downloads/month. Major update: Promptfoo is now part of OpenAI (per the README banner); it remains open source and MIT-licensed, but its roadmap under a frontier lab bears watching.
What it can do
- **YAML evals**: prompts × models × test cases with assertions
- **Red teaming**: automated vulnerability scanning + security reports
- **Model comparison**: OpenAI, Anthropic, Azure, Bedrock, Ollama, and more
- **CI/CD**: gate deploys on eval scores; PR code scanning for LLM security
- **Local-first**: evals run on your machine, results in a web viewer
- **Node.js library**: programmatic use beyond the CLI
Setup
```sh
npm install -g promptfoo
promptfoo init --example getting-started
cd getting-started && promptfoo eval && promptfoo view
```
Set provider keys (e.g. OPENAI_API_KEY). Also via brew or pip install promptfoo. Docs: promptfoo.dev/docs.
Prerequisites
Node.js 18+ (or use npx/brew/pip). LLM provider API keys for the models you evaluate.
Use cases
Works with
Trust score — 9/10
Scored 9/10 (verified tier, metrics-v1). Evidence: 25,942 GitHub stars (log-scaled vs catalog peers); blended with ~3,515,995 monthly package downloads; last push 0d ago; 81 releases/12mo; 698 open issues. Manual review: developer-first eval/red-team CLI; 25.9k stars, 3.5M npm/mo; local-first evals; BUT now part of OpenAI — roadmap independence under a frontier lab is an open question. License: MIT.
★ 25,942 GitHub stars
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 8.6/10
- Maintenance · recency, releases, issue load
- 8.9/10
- Openness · license + docs
- 10.0/10
- Manual review · sanity read + red-flag check
- 8.0/10
- · 25,942 GitHub stars (log-scaled vs catalog peers)
- · blended with ~3,515,995 monthly package downloads
- · last push 0d ago; 81 releases/12mo; 698 open issues
- · manual review: developer-first eval/red-team CLI; 25.9k stars, 3.5M npm/mo; local-first evals; BUT now part of OpenAI — roadmap independence under a frontier lab is an open question
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.