llama-swap

MIT one-binary proxy that hot-swaps local models on demand

AI Agents / Agents · OSS/MIT · 6k

Reviewed 2026-10-05 · last push 2026-10-15 · repo checked 2026-10-19

What's Good

MIT (mostlygeek/llama-swap, release v262). Go, zero dependencies: one binary + one YAML config. Covers chat/completions, responses, embeddings, audio, images, `v1/messages`, rerank; `ttl` auto-unload, profiles, concurrent-model swap matrix, API keys, `/metrics` for Prometheus, and a web UI with playground, token metrics, and log streaming. Docker unified images (CUDA 12/13, Vulkan), Homebrew, WinGet. ★~5.8k.

The Catch

It is a process manager and proxy, not an inference engine — you still configure each upstream server, and a swap means a cold model load (seconds to minutes on big GGUFs). Config is the product: wrong `cmd` flags fail per model. Vs ollama (bundled runtime + model registry) and vllm (serving engine): llama-swap is the switchboard in front of whatever runtimes you already trust.

Verdict

MIT local-model switchboard when agents need many llama.cpp/vLLM models behind one endpoint on limited VRAM — skip if Ollama alone covers your models.

Embed

Maintainer? Add the badge to your README. It is free, links to this review, and there is no paid version.

Markdown

[![Reviewed on Stack Gems](https://stackgems.com/badge/llama-swap.svg)](https://stackgems.com/gems/llama-swap)

HTML

<a href="https://stackgems.com/gems/llama-swap"><img src="https://stackgems.com/badge/llama-swap.svg" alt="llama-swap reviewed on Stack Gems" height="20"></a>

AI Agents →

Related