Best local LLM and inference tools (2026)

1. 1.OllamaMIT local model runner — free forever; Pro cloud $20/mo

   Free / OSS · Freemium · 184k GitHub stars

   VerdictMIT local runner. Free on device; Pro is optional cloud.

   CatchModel runner, not a coding agent. Distinct from vLLM (server serving) and LM Studio (GUI desktop app). You still pick models, prompts, and the agent layer on top.

2. 2.vLLMApache-2.0 high-throughput LLM inference and serving engine

   Free / OSS · Apache-2.0 · 94.1k GitHub stars

   VerdictApache-2.0 LLM serving. Free forever software; the bill is GPUs and the cluster, not a license key.

   CatchVs Ollama (local model runner UX): vLLM is a serving/inference engine, not a laptop chat app. Expect CUDA/ROCm/TPU ops, driver versions, and multi-GPU reality. CPU-only is not the point. You bring hardware and ops.

3. 3.KevApache-2.0 tiny Jev-like decision models you train and serve locally

   Free / OSS · Apache-2.0 · 9.3k GitHub stars

   VerdictApache-2.0 self-host Jev-like decision models. Keep when you want typed agent decisions without paying per LLM-judge call; budget GPU/RAM honestly.

   CatchVs jevals/TypeSafe Jev cloud and ordinary LLM-as-judge evals: Kev is the self-host decision-model family + training repo, not OpenLayer's jevals product wrapper and not a free hosted Jev API. 4B/9B want serious local VRAM/RAM (README cites ~32 GB Mac bf16 class for larger models); first serve downloads base + adapter weights. Decision quality still depends on your question design and calibration — local does not mean calibrated for your domain.

4. 4.ReefApache-2.0 infra for continually self-improving agents

   Free / OSS · Apache-2.0 · 7.6k GitHub stars

   VerdictApache-2.0 continual-improvement infra when agents must learn and redeploy in a loop — not an eval library and not a coding CLI.

   CatchVs Ragas / Phoenix: those are eval libs; Reef is the continual-improvement loop infra. Vs Pi / OpenCode: coding harnesses, not self-improve delivery planes. Research-infra heat — expect ops weight beyond a simple CLI install.

5. 5.MagnitudeApache-2.0 local inference desktop+CLI — consumer profiler for agents

   Free / OSS · Apache-2.0 · 7.2k GitHub stars

   VerdictApache local inference when agents need hardware-aware models on the desk — download magnitude.dev; ignore registry name twins and the ★ spike.

   CatchDo NOT install bare npm/crates `magnitude` — name collision with unrelated packages; get the product from magnitude.dev (and the magnitudedev org), not a random registry hit. ★5,479 in about 21 weeks (~261 a week): judge it on the profiler + agent wiring, not the star count. Vs Ollama / llmfit / vLLM: Ollama is the simple local runner; llmfit sizes models to hardware; vLLM is throughput serving — Magnitude’s catch is consumer profiler + agent harness wiring in one desktop/CLI.

6. 6.vLLM Semantic RouterApache-2.0 Mixture-of-Models router — pick or compose the model path per request

   Free / OSS · Apache-2.0 · 6.2k GitHub stars

   VerdictApache MoM router when one endpoint must route across many models and hardware by intent and cost — overkill if LiteLLM fallbacks already cover you.

   CatchPlatform, not a one-file proxy: it coordinates alongside Envoy and your model servers, so budget real ops time; the docs are written for heterogeneous fleets (cloud/data center/edge) more than a two-model hobby setup. Install is a `curl | bash` script (pip/uv paths exist — prefer those if you audit installs). Vs litellm / bifrost (multi-provider gateways) and weave-router (ELv2 embedder router): this is Apache policy-driven semantic routing that composes model paths.

7. 7.llama-swapMIT one-binary proxy that hot-swaps local models on demand

   Free / OSS · OSS/MIT · 6k GitHub stars

   VerdictMIT local-model switchboard when agents need many llama.cpp/vLLM models behind one endpoint on limited VRAM — skip if Ollama alone covers your models.

   CatchIt is a process manager and proxy, not an inference engine — you still configure each upstream server, and a swap means a cold model load (seconds to minutes on big GGUFs). Config is the product: wrong `cmd` flags fail per model. Vs ollama (bundled runtime + model registry) and vllm (serving engine): llama-swap is the switchboard in front of whatever runtimes you already trust.

8. 8.NVIDIA DynamoApache-2.0 datacenter-scale LLM inference orchestration above vLLM, SGLang and TensorRT-LLM

   Free / OSS · Apache-2.0 · 8.3k GitHub stars

   VerdictApache-2.0 cluster-level LLM serving when you run vLLM/SGLang/TRT-LLM across many GPUs — free, but overkill below multi-GPU scale and benchmarks are vendor-run.

   CatchBuilt for multi-GPU, multi-node serving — the README itself says one model on one GPU only needs the engine (see vLLM, or llama-swap for a local model switcher). Realistically NVIDIA hardware and Kubernetes; the headline speedups are vendor and partner benchmarks on GB200/GB300-class racks, so measure on your own GPUs. Lots of moving parts (frontend, router, workers, planner, discovery backend) and a fast release cadence: pin a version. Vs vLLM Semantic Router (model choice per request) and plain vLLM: Dynamo is the cluster scheduler, not a model router or an engine.

9. 9.Rapid-MLXApache-2.0 local LLM server for Apple Silicon — OpenAI + Anthropic APIs, tool-call parsing tuned for coding agents

   Free / OSS · Apache-2.0 · 4k GitHub stars

   VerdictThe one to try if you run coding agents against local models on a Mac and tool calls keep breaking — free and measured, but Mac-only and benchmarks are the project's own.

   CatchMac and Apple Silicon only. Speed claims come from the project itself on specific Qwen models; the README notes a long-prompt agent turn was close to a tie and a dense 12B model was no faster than Ollama. Model licenses are separate, so check the model card before commercial use. Young (first commit February 2026) and moving fast. Vs Ollama (cross-platform, bigger ecosystem) and oMLX: pick Rapid-MLX when the job is agent tool calls on a Mac.

10. 10.llama.cppMIT local LLM inference in C/C++ on almost any hardware

    Free / OSS · OSS/MIT · 132k GitHub stars

    VerdictThe default local LLM engine — free, runs nearly anywhere, but you own the model choice and the tuning.

    CatchIt runs GGUF models, so you convert or download quantized weights, and model licences are separate — check the model card before commercial use. Lots of knobs: speed depends on backend, quant and offload settings, so expect to tune. Vs Ollama (friendlier model manager on similar ground): pick llama.cpp when you want the engine and its flags directly. Vs vLLM or SGLang: those are GPU serving engines for many concurrent users; llama.cpp is the one for laptops, edge boxes and single-user servers.

Also reviewed (5)

• SGLang — Apache-2.0 high-throughput LLM and multimodal serving
• llm-d — Apache-2.0 Kubernetes stack for distributed LLM inference
• uzu — MIT on-device inference engine for Apple Silicon apps
• TokenSpeed — MIT LLM inference engine built for agent workloads
• ComfyUI — GPL-3.0 node-graph engine and API for local image, video and audio models