GPTCache
Open-source semantic cache that cuts LLM cost and latency by reusing similar past answers.
About
GPTCache is an open-source semantic caching layer built by the engineers at Zilliz, born out of their OSSChat project's struggle with ChatGPT costs and latency. Unlike exact-match caches, it matches semantically similar queries using embeddings, so rephrased questions can hit the cache.
The architecture is a modular pipeline: a pre-embedding step normalizes prompts, an embedding function vectorizes them, a data manager searches across scalar stores like SQLite or PostgreSQL plus vector stores like Milvus, FAISS, Qdrant, or Redis, and a similarity evaluation scores candidates before returning a cached response. Third-party evaluations report 60 to 70 percent cache hit rates with over 92 percent positive-hit rates at a 0.8 cosine threshold, and up to 70 percent API cost savings for repetitive query workloads. It integrates with LangChain and supports custom embedding functions, similarity evaluation, storage, and eviction policies.
What it can do
- Semantic similarity matching instead of exact keys
- Pluggable embeddings, vector stores, and eviction policies
- SQLite, PostgreSQL, Milvus, FAISS, Qdrant, Redis backends
- LangChain integration
- Cache-hit metrics and health checks
- Mock responses for test environments
Setup
pip install gptcache, configure an embedding function and vector store, then wrap LLM calls with the cache.
Prerequisites
Python environment; an embedding source and a scalar/vector store backend.
Use cases
Works with
Trust score — 7/10
Scored 7/10 (metrics tier, metrics-v1). Evidence: Verified via vendor site/docs/GitHub and reputable coverage, wave 91; Trending score 74/100 from researcher assessment.
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.