LMMS-Eval
Unified evaluation harness for large multimodal models.
About
lmms-eval is an open-source evaluation framework for large multimodal models, built by the EvolvingLMMs-Lab community in the design tradition of EleutherAI's lm-evaluation-harness. It brings consistent, reproducible benchmarking to vision-language models across image, video, and audio tasks, with standardized protocols, shareable configurations, and a large catalog of supported tasks.\n\nModel authors use it to run dozens of multimodal benchmarks through one interface, and it ships tooling for adding new models and tasks without rewriting boilerplate. It is installed from PyPI as lmms-eval and has become the community default for comparing open multimodal models.
What it can do
- One-for-all multimodal evaluation across text, image, video, and audio
- Large catalog of supported benchmark tasks with standardized protocols
- Reproducible benchmarking with shareable configurations and JSONL caching
- Plugin-style integration for adding new models and tasks
- Throughput metrics and result-check tooling for paper reproduction
Setup
Run pip install lmms-eval or clone the GitHub repo and install in editable mode for development.
Prerequisites
Python 3.12 recommended and the model implementations you want to benchmark.
Use cases
Works with
Trust score — 7/10
Scored 7/10 (metrics tier, metrics-v1). Evidence: Verified via vendor site/docs/GitHub and reputable coverage, wave 100; Trending score 72/100 from researcher assessment.
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.