# llama.cpp

> MIT local LLM inference in C/C++ on almost any hardware

**Category:** AI Agents  
**Pricing:** open-source  
**URL:** https://llama.app  
**GitHub:** https://github.com/ggml-org/llama.cpp  
**Added:** 2026-10-13

---

## The Hook

llama.cpp is the C/C++ inference engine underneath much of the local-model world: install it, pull a GGUF model from Hugging Face, and `llama serve` gives you an OpenAI-compatible API plus a built-in web UI.

## What's Good

MIT (ggml-org/llama.cpp), release v0.6.0, plus nightly builds. One-line installers (curl or PowerShell), prebuilt binaries, Docker, or build from source. Plain C/C++ with no dependencies; Apple Silicon is first-class via Metal, plus CUDA, HIP (AMD), Vulkan, SYCL, WebGPU and more. 1.5- to 8-bit quantization, and CPU+GPU hybrid inference for models bigger than your VRAM. Handles vision models too, and GBNF grammars constrain output. ★~131k.

## The Catch

It runs GGUF models, so you convert or download quantized weights, and model licences are separate — check the model card before commercial use. Lots of knobs: speed depends on backend, quant and offload settings, so expect to tune. Vs Ollama (friendlier model manager on similar ground): pick llama.cpp when you want the engine and its flags directly. Vs vLLM or SGLang: those are GPU serving engines for many concurrent users; llama.cpp is the one for laptops, edge boxes and single-user servers.

## Verdict

The default local LLM engine — free, runs nearly anywhere, but you own the model choice and the tuning.