AWQ
Activation-aware 4-bit weight quantization for fast on-device LLM inference.
About
AWQ (Activation-Aware Weight Quantization) is an LLM compression method from MIT that protects the small fraction of weight channels tied to large activation magnitudes, since those cause most quantization error. A mathematical scaling transformation shields them before quantization, avoiding the costly Hessian computation that GPTQ relies on.
It won Best Paper at MLSys 2024 and has become the dominant approach for INT4 on-device deployment, with kernels in vLLM, TensorRT-LLM, and AMD's ROCm stack. The companion TinyChat engine demonstrated more than 3x speedup over FP16 Hugging Face inference on desktop and mobile GPUs.
What it can do
- 4-bit weight-only quantization
- No Hessian computation required
- Hardware-friendly uniform INT4 layout
- Kernels for NVIDIA, AMD, Apple, and mobile GPUs
Setup
Clone github.com/mit-han-lab/llm-awq or use AWQ kernels built into vLLM, TensorRT-LLM, and llama.cpp forks.
Prerequisites
Python with PyTorch, CUDA or compatible hardware, and a calibration text sample for activation statistics.
Use cases
Works with
Trust score — 8/10
Scored 8/10 (verified tier, metrics-v1). Evidence: Verified via vendor site/docs/GitHub and reputable coverage, wave 76; Trending score 78/100 from researcher assessment.
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.