mistral.rs
Blazing-fast LLM inference in Rust with an OpenAI-compatible API.
About
mistral.rs is a fast, flexible LLM inference engine written in Rust on top of Hugging Face Candle, maintained by Eric Buehler. It runs text, vision, video, and audio models with GGUF, GPTQ, AWQ, and ISQ quantization, FlashAttention, paged attention, and multi-GPU tensor parallelism.
Any Hugging Face model loads with zero config, and it exposes an OpenAI-compatible HTTP API plus Rust and Python SDKs, a built-in web UI, and agentic features like tool calling, web search, and an MCP client. Hardware-aware tuning picks the best quantization for your machine.
With over 6,000 GitHub stars and an MIT license, it is a popular choice for embedding local inference directly into Rust or Python applications.
What it can do
- Zero-config Hugging Face model loading
- GGUF/GPTQ/AWQ/ISQ quantization
- OpenAI-compatible API plus Rust/Python SDKs
- Tool calling, web search, and MCP client
Setup
Install via cargo (mistralrs crate) or pip (mistralrs), then run mistralrs run -m <hf-model>.
Prerequisites
Rust toolchain or Python 3.10+; a Hugging Face model id.
Use cases
Works with
Trust score — 8/10
Scored 8/10 (verified tier, metrics-v1). Evidence: Verified via vendor site/docs/GitHub and reputable coverage, wave 122; Trending score 76/100 from researcher assessment.
★ 6,000 GitHub stars
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.