bitsandbytes
k-bit quantization primitives for PyTorch: 8-bit optimizers, LLM.int8(), QLoRA.
About
bitsandbytes is the quantization library that made big-model inference and training accessible: 8-bit optimizers (32-bit performance at a fraction of the memory), LLM.int8() (8-bit inference without quality loss via vector-wise quantization with 16-bit outlier handling), and QLoRA (4-bit training with trainable LoRA adapters). Paper-backed (Dettmers et al., QLoRA arXiv 2305.14314) and the engine underneath Transformers' and PEFT's quantization features. MIT, 8.5k stars, 3.4M PyPI/month, with support for NVIDIA, AMD, Intel, Gaudi, and Apple accelerators. You rarely call it directly — it powers the frameworks you do call.
What it can do
- **LLM.int8()**: 8-bit inference, half the memory, no quality loss
- **QLoRA 4-bit**: fine-tune 65B-class models on a single GPU
- **8-bit optimizers**: block-wise quantization for training memory savings
- **Broad hardware**: NVIDIA, AMD, Intel XPU, Gaudi, Apple Metal
- **Framework glue**: powers quantization in Transformers, PEFT, Diffusers
- **Paper-backed**: QLoRA, LLM.int8(), 8-bit optimizer publications
Setup
```bash
pip install bitsandbytes
```
```python
from transformers import AutoModelForCausalLM
# 8-bit inference via Transformers integration
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct", load_in_8bit=True, device_map="auto")
```
Usually consumed via transformers/peft quantization flags. Docs: huggingface.co/docs/bitsandbytes.
Prerequisites
Python 3.10+; PyTorch 2.4+. Accelerator: NVIDIA/AMD/Intel/GPU or Apple silicon (see support matrix).
Use cases
Works with
Trust score — 8/10
Scored 8/10 (verified tier, metrics-v1). Evidence: 8,523 GitHub stars (log-scaled vs catalog peers); blended with ~3,415,159 monthly package downloads; last push 35d ago; 8 releases/12mo; 97 open issues. Manual review: quantization primitives (LLM.int8, QLoRA, 8-bit optimizers); paper-backed; broad accelerator support; 3.4M PyPI/mo; infrastructure — used via frameworks. License: MIT.
★ 8,523 GitHub stars
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 8.0/10
- Maintenance · recency, releases, issue load
- 7.9/10
- Openness · license + docs
- 10.0/10
- Manual review · sanity read + red-flag check
- 9.0/10
- · 8,523 GitHub stars (log-scaled vs catalog peers)
- · blended with ~3,415,159 monthly package downloads
- · last push 35d ago; 8 releases/12mo; 97 open issues
- · manual review: quantization primitives (LLM.int8, QLoRA, 8-bit optimizers); paper-backed; broad accelerator support; 3.4M PyPI/mo; infrastructure — used via frameworks
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.