llama.cpp

MIT local LLM inference in C/C++ on almost any hardware

AI Agents / Local Inference · OSS/MIT · 132k

Reviewed 2026-10-13 · last push 2026-10-19 · repo checked 2026-10-19

What's Good

MIT (ggml-org/llama.cpp), release v0.6.0, plus nightly builds. One-line installers (curl or PowerShell), prebuilt binaries, Docker, or build from source. Plain C/C++ with no dependencies; Apple Silicon is first-class via Metal, plus CUDA, HIP (AMD), Vulkan, SYCL, WebGPU and more. 1.5- to 8-bit quantization, and CPU+GPU hybrid inference for models bigger than your VRAM. Handles vision models too, and GBNF grammars constrain output. ★~131k.

The Catch

It runs GGUF models, so you convert or download quantized weights, and model licences are separate — check the model card before commercial use. Lots of knobs: speed depends on backend, quant and offload settings, so expect to tune. Vs Ollama (friendlier model manager on similar ground): pick llama.cpp when you want the engine and its flags directly. Vs vLLM or SGLang: those are GPU serving engines for many concurrent users; llama.cpp is the one for laptops, edge boxes and single-user servers.

Verdict

The default local LLM engine — free, runs nearly anywhere, but you own the model choice and the tuning.

Embed

Maintainer? Add the badge to your README. It is free, links to this review, and there is no paid version.

Markdown

[![Reviewed on Stack Gems](https://stackgems.com/badge/llama-cpp.svg)](https://stackgems.com/gems/llama-cpp)

HTML

<a href="https://stackgems.com/gems/llama-cpp"><img src="https://stackgems.com/badge/llama-cpp.svg" alt="llama.cpp reviewed on Stack Gems" height="20"></a>

AI Agents →

Related