Skip to content
← Tools / Skills
Toolggml-org/10 trustVerified
llama.cpp logo

llama.cpp

Run LLMs locally with minimal setup and state-of-the-art performance on everyday hardware.

Visit official site →

About

llama.cpp is the reference C/C++ inference engine for large language models (and vision models), built on the ggml tensor library. Its main goal is LLM inference with minimal setup and high performance across a wide range of hardware — Apple Silicon (Metal), NVIDIA (CUDA), AMD (HIP), Intel (SYCL), Vulkan, and plain CPU — locally and in the cloud. It pioneered the GGUF model format and 1.5-to-8-bit integer quantization that made running large models on consumer hardware practical, and it powers most of the local-AI ecosystem (Ollama, LM Studio, and many server wrappers build on it or its conventions).

What it can do

  • Plain C/C++ implementation with **no dependencies** (README-verified). Runs LLMs and VLMs on Apple Silicon (ARM NEON, Accelerate, Metal), x86 (AVX/AVX2/AVX512/AMX), RISC-V, NVIDIA (CUDA), AMD (HIP), Intel (SYCL), Vulkan, Ascend (CANN), Snapdragon (Hexagon), and WebGPU — plus CPU+GPU hybrid inference for models larger than VRAM.
  • **1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization** for reduced memory and faster inference (README-verified). `llama serve` exposes an **OpenAI-compatible HTTP API** with a bundled web UI, GBNF grammar support, and endpoints including `/v1/models` and `/v1/systemone` (decision models, added Oct 2026 per independent coverage).
  • Direct Hugging Face Hub loading (`-hf`), multi-GPU support, Docker images, and RPC-based distributed inference (tools/rpc) round out the deployment story.

Setup

Three install paths (pick one):

  1. Installer (easiest) — visit https://llama.app and follow the instructions.
  1. Pre-built binaries — download from the releases page.
  1. Docker — see docs/docker.md in the repo.
  1. Build from source — clone the repo and follow docs/build.md (CMake; GPU-backend flags like -DGGML_CUDA=ON documented there).

Once installed, run a model straight from the Hugging Face Hub:

```sh

# CLI chat

llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# OpenAI-compatible API server (with built-in web UI)

llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

```

Prerequisites

A GGUF-format model (thousands are on the Hugging Face Hub). For builds from source: CMake, a C/C++ compiler, and optionally a GPU toolkit (CUDA/HIP/oneAPI) matching your hardware. No Python or runtime dependencies — the binaries are self-contained.

Use cases

local-llm-inferencechatbotsagentsragvision-modelsedge-deploymentprivate-offline-ai

Works with

Claude CodeAiderContinueClineOpenCodeCursor

Trust score — 10/10

/10 trustVerified

130,949 GitHub stars, MIT license confirmed in the repo's own LICENSE file, maintained actively (commit pushed 2026-10-12), and the de-facto reference implementation of local LLM inference — an entire ecosystem (Ollama, LM Studio, GGUF) is built on its conventions. Quickstart and key capability claims (no-dependency C/C++, quantization range, backend list, OpenAI-compatible server) verified directly in the README. Only caveat is a governance one: the core team joined Hugging Face in Feb 2026, so maintenance is now HF-employed (project remains MIT/open).

★ 130,949 GitHub stars

How the trust score is built

Full methodology →
Adoption · stars/downloads, log-scaled vs peers
9.4/10
Maintenance · recency, releases, issue load
9.2/10
Openness · license + docs
10.0/10
Manual review · sanity read + red-flag check
10.0/10
  • · 130,949 GitHub stars (log-scaled vs catalog peers)
  • · last push 0d ago; 100 releases/12mo; 2552 open issues
  • · manual review: repo moved ggerganov/llama.cpp → ggml-org/llama.cpp; ggml.ai joined HF (Feb 2026, widely reported); no red flags

Sources

Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.