Tag
Local LLMs
Run or serve models yourself. 23 reviewed tools: 23 free or open source.
Order: free and open source first, then hosted products with a real free tier, then paid-led; catalog order inside each group, with quiet or archived repos last. Not a score, and no tool pays for its place.
23 tools
- ZodMIT TypeScript-first schema validation with static type inference
Free / OSS · OSS/MIT · 44.2k GitHub stars
VerdictMIT TS schemas. Free library; you own the call sites.
CatchTS schema plus static inference; library not SaaS. Bundle size matters at the edge. Not a form UI or API gateway.
- OllamaMIT local model runner — free forever; Pro cloud $20/mo
Free / OSS · Freemium · 184k GitHub stars
VerdictMIT local runner. Free on device; Pro is optional cloud.
CatchModel runner, not a coding agent. Distinct from vLLM (server serving) and LM Studio (GUI desktop app). You still pick models, prompts, and the agent layer on top.
- vLLMApache-2.0 high-throughput LLM inference and serving engine
Free / OSS · Apache-2.0 · 94.1k GitHub stars
VerdictApache-2.0 LLM serving. Free forever software; the bill is GPUs and the cluster, not a license key.
CatchVs Ollama (local model runner UX): vLLM is a serving/inference engine, not a laptop chat app. Expect CUDA/ROCm/TPU ops, driver versions, and multi-GPU reality. CPU-only is not the point. You bring hardware and ops.
- SemIfMIT local/WebGPU semantic conditionals — OpenJev, the former pitch name
Free / OSS · OSS/MIT · 4.8k GitHub stars
VerdictMIT local semantic conditionals. Read logits on frozen open models; do not confuse it with structured output or calibrated refusal.
CatchVs Instructor and structured JSON: SemIf is a logit-read semantic gate, not schema validation or guaranteed JSON. Vs closed TypeSafe Jev: the local/open-model route is the point, but you bring compatible weights, runtime, and calibration discipline — this is not calibrated RLCD. Youth note: ~1.6k★ in ~3 days clears the Panel bar, but the surface is still young.
- NVIDIA Model OptimizerApache-2.0 quantization/pruning/NAS/distill for HF → TensorRT-LLM/vLLM
Free / OSS · Apache-2.0 · 5.4k GitHub stars
VerdictApache-2.0 model-optimization library when you compress HF/PyTorch models for TensorRT-LLM/vLLM/SGLang — skip if you wanted an agent framework.
CatchNVIDIA stack gravity (TensorRT, Megatron, CUDA) — not an agent product. Serious runs need GPU/CUDA reality; CPU toy paths will not match the blog numbers. Ecosystem pull toward NVIDIA deploy targets even though the license is open.
- ReefApache-2.0 infra for continually self-improving agents
Free / OSS · Apache-2.0 · 7.6k GitHub stars
VerdictApache-2.0 continual-improvement infra when agents must learn and redeploy in a loop — not an eval library and not a coding CLI.
CatchVs Ragas / Phoenix: those are eval libs; Reef is the continual-improvement loop infra. Vs Pi / OpenCode: coding harnesses, not self-improve delivery planes. Research-infra heat — expect ops weight beyond a simple CLI install.
- Open WebUISource-available Ollama/OpenAI chat UI — RAG, tools, multi-user
Free / OSS · Open-Core · 156k GitHub stars
VerdictSource-available Ollama/OpenAI chat UI when polished local RAG chat beats building one — branding rules are the Catch vs LibreChat MIT.
CatchBranding lock — removing “Open WebUI” marks is forbidden above ~50 end users / 30 days without written permission or enterprise license (not OSI MIT/Apache). Vs LibreChat (MIT Agents/MCP/skills orchestration UI): LibreChat leans Agents/MCP/skills; Open WebUI leans Ollama-first chat+RAG+admin. You still BYO models, GPU, and upgrade discipline.
- MagnitudeApache-2.0 local inference desktop+CLI — consumer profiler for agents
Free / OSS · Apache-2.0 · 7.2k GitHub stars
VerdictApache local inference when agents need hardware-aware models on the desk — download magnitude.dev; ignore registry name twins and the ★ spike.
CatchDo NOT install bare npm/crates `magnitude` — name collision with unrelated packages; get the product from magnitude.dev (and the magnitudedev org), not a random registry hit. ★5,479 in about 21 weeks (~261 a week): judge it on the profiler + agent wiring, not the star count. Vs Ollama / llmfit / vLLM: Ollama is the simple local runner; llmfit sizes models to hardware; vLLM is throughput serving — Magnitude’s catch is consumer profiler + agent harness wiring in one desktop/CLI.
- OpenSREApache-2.0 self-host SRE agent — Ollama path, hosted quickstart is not free
Free / OSS · Apache-2.0 · 12.2k GitHub stars
VerdictApache self-host SRE agent when you bring Ollama and skip the hosted-model sign-in — not AURA incident team; hosted quickstart is not zero-dollar.
CatchThe quickstart sign-in turns on their hosted model. That is not the free path — self-host with LLM_PROVIDER and set OPENSRE_NO_TELEMETRY=1 if you do not want telemetry. Vs AURA (Mezmo): AURA is an Apache MCP-ready incident harness wired to your observability stack; OpenSRE is the self-host SRE agent with an explicit Ollama provider, not Mezmo specialist incident team. Stars grow fast: judge the self-host loop, not the counter.
- vLLM Semantic RouterApache-2.0 Mixture-of-Models router — pick or compose the model path per request
Free / OSS · Apache-2.0 · 6.2k GitHub stars
VerdictApache MoM router when one endpoint must route across many models and hardware by intent and cost — overkill if LiteLLM fallbacks already cover you.
CatchPlatform, not a one-file proxy: it coordinates alongside Envoy and your model servers, so budget real ops time; the docs are written for heterogeneous fleets (cloud/data center/edge) more than a two-model hobby setup. Install is a `curl | bash` script (pip/uv paths exist — prefer those if you audit installs). Vs litellm / bifrost (multi-provider gateways) and weave-router (ELv2 embedder router): this is Apache policy-driven semantic routing that composes model paths.
- llama-swapMIT one-binary proxy that hot-swaps local models on demand
Free / OSS · OSS/MIT · 6k GitHub stars
VerdictMIT local-model switchboard when agents need many llama.cpp/vLLM models behind one endpoint on limited VRAM — skip if Ollama alone covers your models.
CatchIt is a process manager and proxy, not an inference engine — you still configure each upstream server, and a swap means a cold model load (seconds to minutes on big GGUFs). Config is the product: wrong `cmd` flags fail per model. Vs ollama (bundled runtime + model registry) and vllm (serving engine): llama-swap is the switchboard in front of whatever runtimes you already trust.
- NVIDIA DynamoApache-2.0 datacenter-scale LLM inference orchestration above vLLM, SGLang and TensorRT-LLM
Free / OSS · Apache-2.0 · 8.3k GitHub stars
VerdictApache-2.0 cluster-level LLM serving when you run vLLM/SGLang/TRT-LLM across many GPUs — free, but overkill below multi-GPU scale and benchmarks are vendor-run.
CatchBuilt for multi-GPU, multi-node serving — the README itself says one model on one GPU only needs the engine (see vLLM, or llama-swap for a local model switcher). Realistically NVIDIA hardware and Kubernetes; the headline speedups are vendor and partner benchmarks on GB200/GB300-class racks, so measure on your own GPUs. Lots of moving parts (frontend, router, workers, planner, discovery backend) and a fast release cadence: pin a version. Vs vLLM Semantic Router (model choice per request) and plain vLLM: Dynamo is the cluster scheduler, not a model router or an engine.
- Rapid-MLXApache-2.0 local LLM server for Apple Silicon — OpenAI + Anthropic APIs, tool-call parsing tuned for coding agents
Free / OSS · Apache-2.0 · 4k GitHub stars
VerdictThe one to try if you run coding agents against local models on a Mac and tool calls keep breaking — free and measured, but Mac-only and benchmarks are the project's own.
CatchMac and Apple Silicon only. Speed claims come from the project itself on specific Qwen models; the README notes a long-prompt agent turn was close to a tie and a dense 12B model was no faster than Ollama. Model licenses are separate, so check the model card before commercial use. Young (first commit February 2026) and moving fast. Vs Ollama (cross-platform, bigger ecosystem) and oMLX: pick Rapid-MLX when the job is agent tool calls on a Mac.
- BurnMIT/Apache-2.0 Rust deep-learning framework — train and run on many backends
Free / OSS · Apache-2.0 · 16.1k GitHub stars
VerdictMIT/Apache Rust DL framework when you want training or inference without Python — accept a thinner model zoo and pre-1.0 churn.
CatchFar fewer ready-made models, tutorials, and pretrained weights than PyTorch — ONNX import helps, but not every operator converts. Pre-1.0: expect breaking API changes between releases and pin your version. Smaller community for debugging training issues.
- HAMiApache-2.0 CNCF project that shares and slices GPUs and other accelerators across Kubernetes workloads
Free / OSS · Apache-2.0 · 4.9k GitHub stars
VerdictThe CNCF way to pack many small GPU jobs onto shared cards — validate isolation per accelerator vendor.
CatchIt is a scheduling layer for a GPU cluster you already run, not a model server — you still bring vLLM, Triton or whatever serves the model. How limits are enforced depends on each vendor’s device backend, so isolation and overcommit behaviour differ between NVIDIA and other accelerators; test the limits on your hardware before you rely on them.
- llama.cppMIT local LLM inference in C/C++ on almost any hardware
Free / OSS · OSS/MIT · 132k GitHub stars
VerdictThe default local LLM engine — free, runs nearly anywhere, but you own the model choice and the tuning.
CatchIt runs GGUF models, so you convert or download quantized weights, and model licences are separate — check the model card before commercial use. Lots of knobs: speed depends on backend, quant and offload settings, so expect to tune. Vs Ollama (friendlier model manager on similar ground): pick llama.cpp when you want the engine and its flags directly. Vs vLLM or SGLang: those are GPU serving engines for many concurrent users; llama.cpp is the one for laptops, edge boxes and single-user servers.
- llm-dApache-2.0 Kubernetes stack for distributed LLM inference
Free / OSS · Apache-2.0 · 4.9k GitHub stars
VerdictThe open, Kubernetes-native layer for multi-GPU LLM serving — free, but overkill below cluster scale.
CatchOnly makes sense if you already run Kubernetes and accelerators at scale — for one model on one GPU, a model server alone is enough. The headline gains in the README (for example 3x throughput from prefix-cache-aware routing) come from partner and vendor deployments on high-end GPUs, so measure your own. Still pre-1.0. Vs NVIDIA Dynamo (same layer): llm-d is Kubernetes-native and multi-vendor under the CNCF; Dynamo is NVIDIA-led.
- uzuMIT on-device inference engine for Apple Silicon apps
Free / OSS · OSS/MIT · 2.4k GitHub stars
VerdictThe pick for on-device models inside an iOS or macOS app — MIT and free, but Apple-only for now and tied to its own model format.
CatchApple-first: Linux, Windows and WASM targets are still listed as in progress. It uses its own model format — you download Mirai's converted models or export one yourself with their lalamo tool, so you cannot drop in an arbitrary GGUF. The Rust quick start pulls from the git main branch. Backed by a company (Mirai). Vs Rapid-MLX (a local server for coding agents on a Mac): uzu is an embeddable engine for shipping AI inside iOS and macOS apps.
- EdgeQuakeApache-2.0 GraphRAG engine in Rust, runs on local models
Free / OSS · Apache-2.0 · 2.3k GitHub stars
VerdictLocal-first GraphRAG in Rust — worth a try for private documents, if you can give it a capable extraction model.
CatchPre-1.0 with a small group of contributors (about a dozen), so expect breaking changes. Graph quality depends heavily on the model you use for entity extraction — small local models produce thinner, noisier graphs, and the first local run pulls several GB of models. Vs cognee (a more established memory and knowledge-graph layer for agents): EdgeQuake is the lighter Rust option for local GraphRAG.
- TokenSpeedMIT LLM inference engine built for agent workloads
Free / OSS · OSS/MIT · 2.7k GitHub stars
VerdictA fast-moving inference engine for agent traffic on top-end NVIDIA hardware — benchmark it beside vLLM before committing.
CatchVery early (v0.1.1), so expect breaking changes. The kernel work targets recent NVIDIA data-center GPUs such as Blackwell, so gains on older or consumer cards are unclear. It covers fewer models than vLLM — check yours is supported before you switch.
- UnslothApache-2.0 core for fast local LLM fine-tuning, plus an AGPL desktop studio
Free / OSS · AGPL-3.0 · 78.2k GitHub stars
VerdictThe default way to fine-tune an open model on one GPU — pin the version and know which part is AGPL.
CatchReleases are tagged beta and move fast, so pin versions in training scripts. Fine-tuning still needs a capable GPU for anything beyond small models, and multi-GPU support is newer than the single-GPU path. Watch the license split: the library is Apache-2.0, but the Studio UI is AGPL-3.0. The one-line installer pipes a remote script into your shell, so read it first.
- ComfyUIGPL-3.0 node-graph engine and API for local image, video and audio models
Free / OSS · GPL-3.0 · 139k GitHub stars
VerdictThe default local engine for generative media pipelines — keep custom nodes to ones you trust.
CatchIt is an engine you operate, not a one-click generator: node graphs have a learning curve, and the large custom-node ecosystem is third-party Python you are running with full access to your machine, so vet what you install. Real speed needs a decent GPU. Comfy Cloud and the partner nodes for closed models are paid; the local engine is free. GPL-3.0 matters if you plan to redistribute a modified build.
- MCP for UnityMIT MCP server that lets AI assistants drive the Unity Editor
Free / OSS · OSS/MIT · 15.1k GitHub stars
VerdictThe most complete free bridge between coding agents and the Unity Editor — commit before you prompt.
CatchNeeds Python 3.10+ via uv alongside Unity. An agent with editor access can make a mess of a scene or project, so keep the project in version control and review changes. Maintained and sponsored by Aura, whose paid Unity/Unreal assistant is pitched in the README; the MCP server itself is free. Not affiliated with Unity Technologies.
FAQ
- What is the top free pick among Local LLMs?
- Zod leads the Stack Gems list for Local LLMs. Verdict: MIT TS schemas. Free library; you own the call sites.
- How many Local LLMs does Stack Gems review?
- 23: 23 free or open source. Each review has a verdict and an honest Catch; no tool pays for its spot.
- Which Local LLMs can I self-host for free?
- Zod, vLLM, SemIf, NVIDIA Model Optimizer, Reef, Open WebUI, Magnitude, OpenSRE and 14 more.
- Which Local LLMs have the most GitHub stars?
- Ollama (184k), Open WebUI (156k), ComfyUI (139k), per GitHub on 2026-10-19. Stars measure attention, not fit; read the Catch.
- What is the catch with Zod?
- TS schema plus static inference; library not SaaS.