Tag
Inference tools
10 reviewed tools: 10 free or open source.
Order: free and open source first, then hosted products with a real free tier, then paid-led; catalog order inside each group, with quiet or archived repos last. Not a score, and no tool pays for its place.
10 tools
- colibrìApache-2.0 pure-C zero-deps MoE streamed from disk
Free / OSS · Apache-2.0 · 42.8k GitHub stars
VerdictApache-2.0 pure-C MoE engine. Disk-streamed experts vs Ollama/vLLM; hardware-honest.
CatchVs Ollama/vLLM (local serve / high-throughput GPU serving): colibrì is research-grade MoE disk-streaming for models that do not fit, not the everyday pull-and-chat DX of Ollama or the production serving stack of vLLM. research/HW-heavy — expect fast NVMe, large model containers (hundreds of GB), and tokens/sec that track your disk hierarchy.
- MagnitudeApache-2.0 local inference desktop+CLI — consumer profiler for agents
Free / OSS · Apache-2.0 · 7.2k GitHub stars
VerdictApache local inference when agents need hardware-aware models on the desk — download magnitude.dev; ignore registry name twins and the ★ spike.
CatchDo NOT install bare npm/crates `magnitude` — name collision with unrelated packages; get the product from magnitude.dev (and the magnitudedev org), not a random registry hit. ★5,479 in about 21 weeks (~261 a week): judge it on the profiler + agent wiring, not the star count. Vs Ollama / llmfit / vLLM: Ollama is the simple local runner; llmfit sizes models to hardware; vLLM is throughput serving — Magnitude’s catch is consumer profiler + agent harness wiring in one desktop/CLI.
- DwarfStar 4MIT in-process coding agent — project quants only, shared KV cache
Free / OSS · OSS/MIT · 23.9k GitHub stars
VerdictMIT in-process coding agent when you want the antirez shared-KV quants — not Ollama, not Magnitude, not llmfit.
CatchNot a generic GGUF runner. Project quants only: DeepSeek V4/V4.1, GLM 5.x, Qwen3.8. Vs Ollama / Magnitude / llmfit: Ollama is the general local model runner, Magnitude is the consumer profiler plus agent-harness wiring, llmfit only sizes models to hardware — DwarfStar 4 is the antirez in-process coding agent with a shared KV cache, not a general local runner. Still beta; pin a commit before a fleet bet.
- NVIDIA DynamoApache-2.0 datacenter-scale LLM inference orchestration above vLLM, SGLang and TensorRT-LLM
Free / OSS · Apache-2.0 · 8.3k GitHub stars
VerdictApache-2.0 cluster-level LLM serving when you run vLLM/SGLang/TRT-LLM across many GPUs — free, but overkill below multi-GPU scale and benchmarks are vendor-run.
CatchBuilt for multi-GPU, multi-node serving — the README itself says one model on one GPU only needs the engine (see vLLM, or llama-swap for a local model switcher). Realistically NVIDIA hardware and Kubernetes; the headline speedups are vendor and partner benchmarks on GB200/GB300-class racks, so measure on your own GPUs. Lots of moving parts (frontend, router, workers, planner, discovery backend) and a fast release cadence: pin a version. Vs vLLM Semantic Router (model choice per request) and plain vLLM: Dynamo is the cluster scheduler, not a model router or an engine.
- BurnMIT/Apache-2.0 Rust deep-learning framework — train and run on many backends
Free / OSS · Apache-2.0 · 16.1k GitHub stars
VerdictMIT/Apache Rust DL framework when you want training or inference without Python — accept a thinner model zoo and pre-1.0 churn.
CatchFar fewer ready-made models, tutorials, and pretrained weights than PyTorch — ONNX import helps, but not every operator converts. Pre-1.0: expect breaking API changes between releases and pin your version. Smaller community for debugging training issues.
- llama.cppMIT local LLM inference in C/C++ on almost any hardware
Free / OSS · OSS/MIT · 132k GitHub stars
VerdictThe default local LLM engine — free, runs nearly anywhere, but you own the model choice and the tuning.
CatchIt runs GGUF models, so you convert or download quantized weights, and model licences are separate — check the model card before commercial use. Lots of knobs: speed depends on backend, quant and offload settings, so expect to tune. Vs Ollama (friendlier model manager on similar ground): pick llama.cpp when you want the engine and its flags directly. Vs vLLM or SGLang: those are GPU serving engines for many concurrent users; llama.cpp is the one for laptops, edge boxes and single-user servers.
- SGLangApache-2.0 high-throughput LLM and multimodal serving
Free / OSS · Apache-2.0 · 37.3k GitHub stars
VerdictA top-tier open LLM serving engine beside vLLM — free software, the bill is GPUs; benchmark both on your workload.
CatchA GPU serving engine, not a laptop chat app — expect driver, CUDA/ROCm and multi-GPU ops, and you pay for the hardware. Install currently needs `--prerelease=allow`. Vs vLLM (same lane): both are open, OpenAI-compatible serving engines — benchmark your own model and traffic rather than trusting either side's charts. Vs llama.cpp: SGLang is for many concurrent users on accelerators; llama.cpp is for one machine.
- llm-dApache-2.0 Kubernetes stack for distributed LLM inference
Free / OSS · Apache-2.0 · 4.9k GitHub stars
VerdictThe open, Kubernetes-native layer for multi-GPU LLM serving — free, but overkill below cluster scale.
CatchOnly makes sense if you already run Kubernetes and accelerators at scale — for one model on one GPU, a model server alone is enough. The headline gains in the README (for example 3x throughput from prefix-cache-aware routing) come from partner and vendor deployments on high-end GPUs, so measure your own. Still pre-1.0. Vs NVIDIA Dynamo (same layer): llm-d is Kubernetes-native and multi-vendor under the CNCF; Dynamo is NVIDIA-led.
- uzuMIT on-device inference engine for Apple Silicon apps
Free / OSS · OSS/MIT · 2.4k GitHub stars
VerdictThe pick for on-device models inside an iOS or macOS app — MIT and free, but Apple-only for now and tied to its own model format.
CatchApple-first: Linux, Windows and WASM targets are still listed as in progress. It uses its own model format — you download Mirai's converted models or export one yourself with their lalamo tool, so you cannot drop in an arbitrary GGUF. The Rust quick start pulls from the git main branch. Backed by a company (Mirai). Vs Rapid-MLX (a local server for coding agents on a Mac): uzu is an embeddable engine for shipping AI inside iOS and macOS apps.
- TokenSpeedMIT LLM inference engine built for agent workloads
Free / OSS · OSS/MIT · 2.7k GitHub stars
VerdictA fast-moving inference engine for agent traffic on top-end NVIDIA hardware — benchmark it beside vLLM before committing.
CatchVery early (v0.1.1), so expect breaking changes. The kernel work targets recent NVIDIA data-center GPUs such as Blackwell, so gains on older or consumer cards are unclear. It covers fewer models than vLLM — check yours is supported before you switch.
FAQ
- What is the top free pick among Inference tools?
- colibrì leads the Stack Gems list for Inference tools. Verdict: Apache-2.0 pure-C MoE engine. Disk-streamed experts vs Ollama/vLLM; hardware-honest.
- How many Inference tools does Stack Gems review?
- 10: 10 free or open source. Each review has a verdict and an honest Catch; no tool pays for its spot.
- Which Inference tools can I self-host for free?
- colibrì, Magnitude, DwarfStar 4, NVIDIA Dynamo, Burn, llama.cpp, SGLang, llm-d and 2 more.
- Which Inference tools have the most GitHub stars?
- llama.cpp (132k), colibrì (42.8k), SGLang (37.3k), per GitHub on 2026-10-19. Stars measure attention, not fit; read the Catch.
- What is the catch with colibrì?
- Vs Ollama/vLLM (local serve / high-throughput GPU serving): colibrì is research-grade MoE disk-streaming for models that do not fit, not the everyday pull-and-chat DX of Ollama or the production serving stack of vLLM.