LLMLingua
Microsoft's prompt compression toolkit squeezing prompts up to 20x with small models.
About
LLMLingua is a prompt compression library from Microsoft Research, MIT licensed, that uses small language model perplexity scoring to identify and remove low-information tokens from long prompts. It reports up to 20x compression with minimal quality loss, plus 20 to 30 percent reductions in output length, saving API costs for token-heavy workloads.
Compression is extractive and robustness-oriented: no training of the target LLM is needed, and compressed prompts work directly with black-box APIs like GPT-4 and Claude. The toolkit includes LongLLMLingua for long-context scenarios and can also compress KV-cache to speed inference. Caveat: the upstream repo has been quiet since v0.2.2 in April 2024, and related research continues under the SecurityLingua line, so plan for a possible future migration.
What it can do
- Token-level prompt compression up to 20x
- LongLLMLingua for long-context documents
- KV-cache compression for faster inference
- Works with black-box LLM APIs
- Question-aware compression modes
- MIT licensed
Setup
pip install llmlingua, then use PromptCompressor with a target token budget.
Prerequisites
Python environment; a small compressor model such as GPT-2 or phi-2 downloaded on first use.
Use cases
Works with
Trust score — 7/10
Scored 7/10 (metrics tier, metrics-v1). Evidence: Verified via vendor site/docs/GitHub and reputable coverage, wave 91; Trending score 66/100 from researcher assessment.
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.