LLM Compressor
First-party quantization and sparsity toolkit for vLLM deployment.
About
LLM Compressor is the vLLM Project's official library for compressing large language models before serving them with vLLM. Maintained jointly with Red Hat AI, it applies one-shot quantization recipes, GPTQ, AWQ, SmoothQuant, FP8, and structured sparsity, then emits models in the compressed-tensors format that vLLM loads natively.\n\nThe workflow is recipe-based: you declare modifiers for sparsity and quantization, run the oneshot compression against a calibration dataset, and get a deployment-ready checkpoint tuned for the GPU architecture you target, from Turing up through Blackwell. vLLM's own documentation recommends it as the starting point for quantization, and it has become the default path for producing efficient serving checkpoints in that ecosystem.
What it can do
- One-shot quantization recipes with GPTQ, AWQ, FP8, and more
- Structured 2:4 sparsity for hardware-accelerated inference
- compressed-tensors output that vLLM loads natively
- Architecture-aware schemes from Turing through Blackwell
- Jointly maintained by Red Hat AI and the vLLM Project
Setup
Clone the vllm-project/llm-compressor repo and pip install -e ., then run a compression recipe script.
Prerequisites
Python, PyTorch, an NVIDIA GPU, and a Hugging Face model to compress.
Use cases
Works with
Trust score — 8/10
Scored 8/10 (verified tier, metrics-v1). Evidence: Verified via vendor site/docs/GitHub and reputable coverage, wave 100; Trending score 80/100 from researcher assessment.
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.