AutoGPTQ
Easy-to-use GPTQ quantization library for squeezing LLMs to 4-bit.
About
AutoGPTQ is the community-standard Python library for applying the GPTQ algorithm to large language models. It takes a full-precision model and calibration data, and produces 4-bit, 3-bit, or even 2-bit weight-quantized versions that run dramatically faster with a fraction of the memory, making models like 70B parameter ones runnable on consumer or single-datacenter GPUs.\n\nThe API is deliberately simple: a BaseQuantizeConfig declares bits, group size, and dampening, then AutoGPTQForCausalLM quantizes and saves the model in safetensors format, ready to load through Hugging Face Transformers. It supports multiple inference backends including ExLlamaV2 for speed, Marlin on Ampere-and-newer GPUs, and Triton on Linux, and its output format is understood by most serving stacks.
What it can do
- Weight-only quantization down to 2, 3, 4, or 8 bits with calibration
- User-friendly API built directly on Hugging Face Transformers
- Multiple backends: ExLlamaV2, Marlin, Triton, and CUDA
- Safetensors output loadable by most serving runtimes
- Near-FP16 accuracy at roughly a quarter of the memory
Setup
Install from PyPI or the GitHub repo, prepare calibration data, and quantize with a BaseQuantizeConfig script.
Prerequisites
Python, PyTorch, and an NVIDIA GPU for calibration and inference.
Use cases
Works with
Trust score — 8/10
Scored 8/10 (verified tier, metrics-v1). Evidence: Verified via vendor site/docs/GitHub and reputable coverage, wave 100; Trending score 76/100 from researcher assessment.
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.