LMDeploy
High-throughput LLM/VLM compression, deployment, and serving toolkit.
About
LMDeploy is InternLM's toolkit for compressing, deploying, and serving large language and vision-language models. It ships two inference engines (TurboMind for max throughput, PyTorch for flexibility), 4-bit/FP8/MXFP4 quantization, an OpenAI-compatible API server, and features aimed at production scale: prefill-decode disaggregation, CUDA graphs, and multi-GPU vision-model balancing. It tracks new model releases aggressively (DeepSeek V3/R1, Qwen3.5, InternVL) and supports NVIDIA GPUs plus Huawei Ascend.
What it can do
- **Two engines**: TurboMind (throughput) and PyTorch (flexibility/debuggability)
- **Quantization**: 4-bit weight-only, FP8, MXFP4, plus llm-compressor integration
- **OpenAI-compatible server**: drop-in `serve api_server` for chat/completions and tool calling
- **PD disaggregation**: split prefill/decode across workers for long-context serving
- **VLM support**: InternVL, Qwen-VL, LLaVA families with multi-GPU balancing
- **Broad hardware**: NVIDIA CUDA graphs plus Huawei Ascend support
Setup
```bash
pip install lmdeploy
```
```bash
# Serve a model with an OpenAI-compatible endpoint
lmdeploy serve api_server Qwen/Qwen2.5-7B-Instruct
```
Then call it like any OpenAI API at http://localhost:23333/v1. See the quickstart at lmdeploy.readthedocs.io for quantization (lmdeploy lite) and engine choices.
Prerequisites
Python 3.9+; NVIDIA GPU with recent CUDA (or Ascend NPU). Several GB of VRAM/RAM depending on model size and quantization.
Use cases
Works with
Trust score — 8/10
Scored 8/10 (verified tier, metrics-v1). Evidence: 8,104 GitHub stars (log-scaled vs catalog peers); blended with ~23,478 monthly package downloads; last push 4d ago; 15 releases/12mo; 613 open issues. Manual review: serious production inference toolkit (TurboMind+PyTorch, quantization, PD disaggregation); heavyweight NVIDIA-centric install; vendor perf claims are hardware-specific. License: Apache-2.0.
★ 8,104 GitHub stars
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 7.2/10
- Maintenance · recency, releases, issue load
- 8.2/10
- Openness · license + docs
- 10.0/10
- Manual review · sanity read + red-flag check
- 9.0/10
- · 8,104 GitHub stars (log-scaled vs catalog peers)
- · blended with ~23,478 monthly package downloads
- · last push 4d ago; 15 releases/12mo; 613 open issues
- · manual review: serious production inference toolkit (TurboMind+PyTorch, quantization, PD disaggregation); heavyweight NVIDIA-centric install; vendor perf claims are hardware-specific
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.