llama-swap
Hot-swap multiple local models behind one OpenAI-compatible server.
About
llama-swap is a single-binary Go proxy that sits in front of llama.cpp, vLLM, and other local inference servers and switches between models on demand. When a request arrives for a model that is not loaded, it spins the model up and tears down the old one, so a single machine can serve dozens of models through one endpoint without manual juggling.\n\nIt exposes OpenAI and Anthropic compatible endpoints for chat, completions, embeddings, and audio, plus custom endpoints for health checks, manual unload, and a small web UI. The maintainer describes it as used by thousands of people for local AI workflows, and configuration is one YAML file with no dependencies.
What it can do
- Automatic on-demand swapping between multiple local models
- OpenAI and Anthropic compatible API surface including audio endpoints
- Per-model server settings, preload hooks, and idle unload timeouts
- Groups for running several models at once when memory allows
- Web UI, remote log monitoring, and direct upstream access endpoints
Setup
Download the prebuilt binary, Docker image, or Homebrew package, write a config YAML listing your models, and run the server.
Prerequisites
A working llama.cpp or vLLM server install and GGUF or safetensors models.
Use cases
Works with
Trust score — 8/10
Scored 8/10 (verified tier, metrics-v1). Evidence: Verified via vendor site/docs/GitHub and reputable coverage, wave 100; Trending score 76/100 from researcher assessment.
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.