{"data":[{"slug":"ollama","name":"Ollama","tagline":"MIT local model runner — free forever; Pro cloud $20/mo","category":"agents","url":"https://ollama.com","github":"https://github.com/ollama/ollama","pricing":"freemium","hook":"Ollama is the local free-forever model runner when you want models on your machine without a coding-agent shell.","whatsGood":"MIT. Pull and run open models locally with a simple CLI/API. Free forever on device; optional Pro cloud at $20/mo. Huge community default for laptop LLM experiments.","theCatch":"Model runner, not a coding agent. Distinct from vLLM (server serving) and LM Studio (GUI desktop app). You still pick models, prompts, and the agent layer on top.","verdict":"MIT local runner. Free on device; Pro is optional cloud.","addedAt":"2026-09-05","featured":false,"domain":"AI Agents","subSpecialty":"Agents","subcategory":"Inference & Local Models","tags":["local-first","free-tier","permissive","coding-agents","local-llm","active","fast-stars"],"capabilities":[],"surfaces":["Hosted"],"ecosystem":["Multi-platform"],"licenseModel":"Freemium","githubStars":183774,"starsPerWeek":1062,"access":"free","archived":false,"repo":{"createdAt":"2023-06-26","pushedAt":"2026-10-18","license":"MIT","archived":false,"checkedAt":"2026-10-19"},"links":{"self":"/api/gems/ollama","html":"/gems/ollama","markdown":"/gems/ollama.md"}},{"slug":"vllm","name":"vLLM","tagline":"Apache-2.0 high-throughput LLM inference and serving engine","category":"agents","url":"https://vllm.ai","github":"https://github.com/vllm-project/vllm","pricing":"open-source","hook":"vLLM is the OpenAI-compatible serving engine when local chat UIs are not enough and you need PagedAttention throughput.","whatsGood":"Apache-2.0. vllm-project/vllm. Continuous batching, PagedAttention, broad model/quantization support, OpenAI-compatible API server. Built for GPU (and growing accelerator) serving at scale — docs at docs.vllm.ai.","theCatch":"Vs Ollama (local model runner UX): vLLM is a serving/inference engine, not a laptop chat app. Expect CUDA/ROCm/TPU ops, driver versions, and multi-GPU reality. CPU-only is not the point. You bring hardware and ops.","verdict":"Apache-2.0 LLM serving. Free forever software; the bill is GPUs and the cluster, not a license key.","addedAt":"2026-09-11","featured":false,"domain":"AI Agents","subSpecialty":"Agents","subcategory":"Inference & Local Models","tags":["self-host","permissive","local-llm","active"],"capabilities":[],"surfaces":["Self-host"],"ecosystem":["Multi-platform"],"licenseModel":"Apache-2.0","githubStars":94095,"starsPerWeek":488,"access":"free","archived":false,"repo":{"createdAt":"2023-02-09","pushedAt":"2026-10-19","license":"Apache-2.0","archived":false,"checkedAt":"2026-10-19"},"links":{"self":"/api/gems/vllm","html":"/gems/vllm","markdown":"/gems/vllm.md"}},{"slug":"kev","name":"Kev","tagline":"Apache-2.0 tiny Jev-like decision models you train and serve locally","category":"agents","url":"https://github.com/jaredpalmer/kev","github":"https://github.com/jaredpalmer/kev","pricing":"open-source","hook":"Kev is a family of small Jev-style decision models (0.8B/4B/9B on Qwen3.5) you can train and serve yourself for yes/no, choice, and score tickets.","whatsGood":"Apache-2.0 (jaredpalmer/kev). Pretrained weights, training code, frozen eval suites, and a local serve path (`uv sync --extra serve`) that speaks a System One-compatible API so TypeSafe's Python SDK can point at your box. CUDA/ROCm/Apple Silicon notes; HF Spaces demo for trying order-sensitivity without an install. Useful when agent loops need cheap structured decisions instead of another LLM judge call.","theCatch":"Vs jevals/TypeSafe Jev cloud and ordinary LLM-as-judge evals: Kev is the self-host decision-model family + training repo, not OpenLayer's jevals product wrapper and not a free hosted Jev API. 4B/9B want serious local VRAM/RAM (README cites ~32 GB Mac bf16 class for larger models); first serve downloads base + adapter weights. Decision quality still depends on your question design and calibration — local does not mean calibrated for your domain.","verdict":"Apache-2.0 self-host Jev-like decision models. Keep when you want typed agent decisions without paying per LLM-judge call; budget GPU/RAM honestly.","addedAt":"2026-09-22","featured":false,"domain":"AI Agents","subSpecialty":"Agents","subcategory":"Inference & Local Models","tags":["self-host","permissive","active","young-repo","fast-stars"],"capabilities":[],"surfaces":["Self-host"],"ecosystem":["Multi-platform"],"licenseModel":"Apache-2.0","githubStars":9295,"starsPerWeek":1997,"access":"free","archived":false,"repo":{"createdAt":"2026-09-17","pushedAt":"2026-10-16","license":"Apache-2.0","archived":false,"checkedAt":"2026-10-19"},"links":{"self":"/api/gems/kev","html":"/gems/kev","markdown":"/gems/kev.md"}},{"slug":"reef","name":"Reef","tagline":"Apache-2.0 infra for continually self-improving agents","category":"agents","url":"https://reefinfra.ai","github":"https://github.com/Human-Agent-Society/reef","pricing":"open-source","hook":"Reef is OSS infrastructure for continually self-improving agents — inference, feedback, learning, and versioned delivery in one loop, not another coding CLI.","whatsGood":"Apache-2.0 (Human-Agent-Society/reef). PyPI `reef-infra`; train weights (Slime/SGLang) or improve harness prompts/rules/skills with versioned delivery. Docs quickstart at reefinfra.ai. ★6,081 from GitHub API 2026-09-27.","theCatch":"Vs Ragas / Phoenix: those are eval libs; Reef is the continual-improvement loop infra. Vs Pi / OpenCode: coding harnesses, not self-improve delivery planes. Research-infra heat — expect ops weight beyond a simple CLI install.","verdict":"Apache-2.0 continual-improvement infra when agents must learn and redeploy in a loop — not an eval library and not a coding CLI.","addedAt":"2026-09-27","featured":false,"domain":"AI Agents","subSpecialty":"Agents","subcategory":"Inference & Local Models","tags":["cli","self-host","permissive","coding-agents","local-llm","active","young-repo","fast-stars"],"capabilities":["cli"],"surfaces":["CLI","Self-host"],"ecosystem":["Multi-platform"],"licenseModel":"Apache-2.0","githubStars":7647,"starsPerWeek":1079,"access":"free","archived":false,"repo":{"createdAt":"2026-08-31","pushedAt":"2026-10-19","license":"Apache-2.0","archived":false,"checkedAt":"2026-10-19"},"links":{"self":"/api/gems/reef","html":"/gems/reef","markdown":"/gems/reef.md"}},{"slug":"magnitude","name":"Magnitude","tagline":"Apache-2.0 local inference desktop+CLI — consumer profiler for agents","category":"agents","url":"https://magnitude.dev","github":"https://github.com/magnitudedev/magnitude","pricing":"open-source","hook":"Magnitude profiles your machine, recommends local models, then downloads, tunes, and runs them — and wires the agent harness you already use to localhost.","whatsGood":"Apache-2.0 (magnitudedev/magnitude). Consumer-hardware inference with hardware profiling, tok/s estimates before download, speculative decoding, and one-click wiring to Pi / OpenCode / Claude Code / Cline / Codex and friends. Desktop + CLI; free offline path once models are local. Download from magnitude.dev.","theCatch":"Do NOT install bare npm/crates `magnitude` — name collision with unrelated packages; get the product from magnitude.dev (and the magnitudedev org), not a random registry hit. ★5,479 in about 21 weeks (~261 a week): judge it on the profiler + agent wiring, not the star count. Vs Ollama / llmfit / vLLM: Ollama is the simple local runner; llmfit sizes models to hardware; vLLM is throughput serving — Magnitude’s catch is consumer profiler + agent harness wiring in one desktop/CLI.","verdict":"Apache local inference when agents need hardware-aware models on the desk — download magnitude.dev; ignore registry name twins and the ★ spike.","addedAt":"2026-09-30","featured":false,"domain":"AI Agents","subSpecialty":"Local Inference","subcategory":"Inference & Local Models","tags":["cli","desktop-app","self-host","local-first","permissive","local-llm","active","young-repo"],"capabilities":["inference","local","agents"],"surfaces":["CLI","Self-host"],"ecosystem":["Multi-platform"],"licenseModel":"Apache-2.0","githubStars":7169,"starsPerWeek":387,"access":"free","archived":false,"repo":{"createdAt":"2026-06-12","pushedAt":"2026-10-19","license":"Apache-2.0","archived":false,"checkedAt":"2026-10-19"},"links":{"self":"/api/gems/magnitude","html":"/gems/magnitude","markdown":"/gems/magnitude.md"}},{"slug":"vllm-semantic-router","name":"vLLM Semantic Router","tagline":"Apache-2.0 Mixture-of-Models router — pick or compose the model path per request","category":"agents","url":"https://vllm-sr.ai","github":"https://github.com/vllm-project/semantic-router","pricing":"open-source","hook":"vLLM Semantic Router sits between your OpenAI/Anthropic-compatible endpoint and your model fleet and decides, per request, which model path to run — direct, specialist, local, cascade, or collaborative — from signals like intent, difficulty, modality, identity, and risk.","whatsGood":"Apache-2.0 (vllm-project/semantic-router, v0.4.0; PyPI `vllm-sr` 0.4.0). Go core in the vLLM org. Stable entrypoints resolve to “recipes” that can attach retrieval, memory, tool filtering, caching, safety checks, and verification. Works with Envoy and existing inference runtimes rather than replacing them; Docker, Kubernetes, and hardware paths documented; public playground. ★~6k.","theCatch":"Platform, not a one-file proxy: it coordinates alongside Envoy and your model servers, so budget real ops time; the docs are written for heterogeneous fleets (cloud/data center/edge) more than a two-model hobby setup. Install is a `curl | bash` script (pip/uv paths exist — prefer those if you audit installs). Vs litellm / bifrost (multi-provider gateways) and weave-router (ELv2 embedder router): this is Apache policy-driven semantic routing that composes model paths.","verdict":"Apache MoM router when one endpoint must route across many models and hardware by intent and cost — overkill if LiteLLM fallbacks already cover you.","addedAt":"2026-10-05","featured":false,"domain":"AI Agents","subSpecialty":"Agents","subcategory":"Inference & Local Models","tags":["self-host","permissive","local-llm","active"],"capabilities":[],"surfaces":["Self-host"],"ecosystem":["Multi-platform"],"licenseModel":"Apache-2.0","githubStars":6213,"starsPerWeek":104,"access":"free","archived":false,"repo":{"createdAt":"2025-08-26","pushedAt":"2026-10-19","license":"Apache-2.0","archived":false,"checkedAt":"2026-10-19"},"links":{"self":"/api/gems/vllm-semantic-router","html":"/gems/vllm-semantic-router","markdown":"/gems/vllm-semantic-router.md"}},{"slug":"llama-swap","name":"llama-swap","tagline":"MIT one-binary proxy that hot-swaps local models on demand","category":"agents","url":"https://github.com/mostlygeek/llama-swap","github":"https://github.com/mostlygeek/llama-swap","pricing":"open-source","hook":"llama-swap puts one OpenAI- and Anthropic-compatible endpoint in front of llama.cpp, vLLM, stable-diffusion.cpp, whisper/audio.cpp and ComfyUI, starting and stopping the right server when a request names a model — so your agents can use many local models without all of them sitting in VRAM.","whatsGood":"MIT (mostlygeek/llama-swap, release v262). Go, zero dependencies: one binary + one YAML config. Covers chat/completions, responses, embeddings, audio, images, `v1/messages`, rerank; `ttl` auto-unload, profiles, concurrent-model swap matrix, API keys, `/metrics` for Prometheus, and a web UI with playground, token metrics, and log streaming. Docker unified images (CUDA 12/13, Vulkan), Homebrew, WinGet. ★~5.8k.","theCatch":"It is a process manager and proxy, not an inference engine — you still configure each upstream server, and a swap means a cold model load (seconds to minutes on big GGUFs). Config is the product: wrong `cmd` flags fail per model. Vs ollama (bundled runtime + model registry) and vllm (serving engine): llama-swap is the switchboard in front of whatever runtimes you already trust.","verdict":"MIT local-model switchboard when agents need many llama.cpp/vLLM models behind one endpoint on limited VRAM — skip if Ollama alone covers your models.","addedAt":"2026-10-05","featured":false,"domain":"AI Agents","subSpecialty":"Agents","subcategory":"Inference & Local Models","tags":["self-host","permissive","local-llm","voice","active"],"capabilities":[],"surfaces":["Self-host"],"ecosystem":["Multi-platform"],"licenseModel":"OSS/MIT","githubStars":6010,"starsPerWeek":56,"access":"free","archived":false,"repo":{"createdAt":"2024-10-04","pushedAt":"2026-10-15","license":"MIT","archived":false,"checkedAt":"2026-10-19"},"links":{"self":"/api/gems/llama-swap","html":"/gems/llama-swap","markdown":"/gems/llama-swap.md"}},{"slug":"nvidia-dynamo","name":"NVIDIA Dynamo","tagline":"Apache-2.0 datacenter-scale LLM inference orchestration above vLLM, SGLang and TensorRT-LLM","category":"agents","url":"https://docs.nvidia.com/dynamo/","github":"https://github.com/ai-dynamo/dynamo","pricing":"open-source","hook":"Dynamo is the orchestration layer above inference engines: it keeps vLLM, SGLang or TensorRT-LLM as the engine and turns a cluster of GPUs into one coordinated, OpenAI-compatible serving system with disaggregated prefill/decode, KV-aware routing, KV cache offload and SLA-driven autoscaling.","whatsGood":"Apache-2.0 (ai-dynamo/dynamo; a few test-data files are MIT). Free path: `pip install ai-dynamo` or pull the prebuilt runtime containers from NVIDIA NGC, then start a frontend plus a vLLM or SGLang worker. KV-aware routing avoids recomputing shared prefixes, the KV Block Manager spills cache to CPU, SSD or object storage, and the Planner right-sizes prefill and decode pools against latency targets. Kubernetes path includes a Gateway API Inference Extension plugin. The 1.0 line adds per-request hints for agentic workloads, and the repo ships agent skills for deploying and debugging Dynamo. Rust core, Python extension points. ★~8.2k.","theCatch":"Built for multi-GPU, multi-node serving — the README itself says one model on one GPU only needs the engine (see vLLM, or llama-swap for a local model switcher). Realistically NVIDIA hardware and Kubernetes; the headline speedups are vendor and partner benchmarks on GB200/GB300-class racks, so measure on your own GPUs. Lots of moving parts (frontend, router, workers, planner, discovery backend) and a fast release cadence: pin a version. Vs vLLM Semantic Router (model choice per request) and plain vLLM: Dynamo is the cluster scheduler, not a model router or an engine.","verdict":"Apache-2.0 cluster-level LLM serving when you run vLLM/SGLang/TRT-LLM across many GPUs — free, but overkill below multi-GPU scale and benchmarks are vendor-run.","addedAt":"2026-10-06","featured":false,"domain":"AI Agents","subSpecialty":"Inference","subcategory":"Inference & Local Models","tags":["self-host","permissive","license-unclear","local-llm","workflows","active"],"capabilities":["inference","llm-serving","kubernetes"],"surfaces":["Self-host"],"ecosystem":["Rust","Python"],"licenseModel":"Apache-2.0","githubStars":8329,"starsPerWeek":98,"access":"free","archived":false,"repo":{"createdAt":"2025-03-03","pushedAt":"2026-10-19","license":"NOASSERTION","archived":false,"checkedAt":"2026-10-19"},"links":{"self":"/api/gems/nvidia-dynamo","html":"/gems/nvidia-dynamo","markdown":"/gems/nvidia-dynamo.md"}},{"slug":"rapid-mlx","name":"Rapid-MLX","tagline":"Apache-2.0 local LLM server for Apple Silicon — OpenAI + Anthropic APIs, tool-call parsing tuned for coding agents","category":"agents","url":"https://rapidmlx.com","github":"https://github.com/raullenchai/Rapid-MLX","pricing":"open-source","hook":"Rapid-MLX runs local models on a Mac through MLX and exposes OpenAI (/v1/chat/completions, /v1/responses) and Anthropic Messages (/v1/messages) endpoints, so Claude Code, Codex-style agents and anything OpenAI-compatible can point at your own machine instead of a cloud API.","whatsGood":"Apache-2.0 (raullenchai/Rapid-MLX). Free path: brew install rapid-mlx (in homebrew-core) or pip install, pull a model, start the server. Focus is reliable tool calling: 27 parser modules with auto-detect fallback, plus continuous batching, a radix prefix cache saved across restarts, and quantized KV cache. The README publishes method and raw data: about 1.5x faster decode than mlx-lm at the median of 18 coding tasks on an M4 Pro Mac mini, up to about 4x on whole-file edits thanks to speculative decoding. A macOS desktop app exists if you want a GUI. ★~3.9k.","theCatch":"Mac and Apple Silicon only. Speed claims come from the project itself on specific Qwen models; the README notes a long-prompt agent turn was close to a tie and a dense 12B model was no faster than Ollama. Model licenses are separate, so check the model card before commercial use. Young (first commit February 2026) and moving fast. Vs Ollama (cross-platform, bigger ecosystem) and oMLX: pick Rapid-MLX when the job is agent tool calls on a Mac.","verdict":"The one to try if you run coding agents against local models on a Mac and tool calls keep breaking — free and measured, but Mac-only and benchmarks are the project's own.","addedAt":"2026-10-07","featured":false,"domain":"AI Agents","subSpecialty":"Agents","subcategory":"Inference & Local Models","tags":["api","self-host","permissive","license-unclear","coding-agents","local-llm","active"],"capabilities":["api"],"surfaces":["API","Self-host"],"ecosystem":["Multi-platform"],"licenseModel":"Apache-2.0","githubStars":3996,"starsPerWeek":118,"access":"free","archived":false,"repo":{"createdAt":"2026-02-25","pushedAt":"2026-10-19","license":"NOASSERTION","archived":false,"checkedAt":"2026-10-19"},"links":{"self":"/api/gems/rapid-mlx","html":"/gems/rapid-mlx","markdown":"/gems/rapid-mlx.md"}},{"slug":"llama-cpp","name":"llama.cpp","tagline":"MIT local LLM inference in C/C++ on almost any hardware","category":"agents","url":"https://llama.app","github":"https://github.com/ggml-org/llama.cpp","pricing":"open-source","hook":"llama.cpp is the C/C++ inference engine underneath much of the local-model world: install it, pull a GGUF model from Hugging Face, and `llama serve` gives you an OpenAI-compatible API plus a built-in web UI.","whatsGood":"MIT (ggml-org/llama.cpp), release v0.6.0, plus nightly builds. One-line installers (curl or PowerShell), prebuilt binaries, Docker, or build from source. Plain C/C++ with no dependencies; Apple Silicon is first-class via Metal, plus CUDA, HIP (AMD), Vulkan, SYCL, WebGPU and more. 1.5- to 8-bit quantization, and CPU+GPU hybrid inference for models bigger than your VRAM. Handles vision models too, and GBNF grammars constrain output. ★~131k.","theCatch":"It runs GGUF models, so you convert or download quantized weights, and model licences are separate — check the model card before commercial use. Lots of knobs: speed depends on backend, quant and offload settings, so expect to tune. Vs Ollama (friendlier model manager on similar ground): pick llama.cpp when you want the engine and its flags directly. Vs vLLM or SGLang: those are GPU serving engines for many concurrent users; llama.cpp is the one for laptops, edge boxes and single-user servers.","verdict":"The default local LLM engine — free, runs nearly anywhere, but you own the model choice and the tuning.","addedAt":"2026-10-13","featured":false,"domain":"AI Agents","subSpecialty":"Local Inference","subcategory":"Inference & Local Models","tags":["api","web-ui","self-host","permissive","local-llm","active","fast-stars"],"capabilities":["inference","local","llm-serving"],"surfaces":["API","Self-host"],"ecosystem":["Multi-platform"],"licenseModel":"OSS/MIT","githubStars":131654,"starsPerWeek":698,"access":"free","archived":false,"repo":{"createdAt":"2023-03-10","pushedAt":"2026-10-19","license":"MIT","archived":false,"checkedAt":"2026-10-19"},"links":{"self":"/api/gems/llama-cpp","html":"/gems/llama-cpp","markdown":"/gems/llama-cpp.md"}},{"slug":"sglang","name":"SGLang","tagline":"Apache-2.0 high-throughput LLM and multimodal serving","category":"agents","url":"https://www.sglang.io","github":"https://github.com/sgl-project/sglang","pricing":"open-source","hook":"SGLang is an open-source serving framework for language, vision-language and diffusion models, tuned for agentic workloads, large-scale serving and RL rollouts.","whatsGood":"Apache-2.0 (sgl-project/sglang), release v0.5.21, hosted by the non-profit LMSYS. Free path: `docker pull lmsysorg/sglang:latest` or `uv pip install sglang`, then pick a launch command from the Cookbook for your model and hardware. Supports NVIDIA, AMD Instinct, Google TPU, Intel GPU/CPU, Apple Silicon via Metal/MLX and Huawei Ascend. Image and video generation (SGLang Diffusion) ships in the same package. Orchestrators such as llm-d, Ray Serve and NVIDIA Dynamo can run it as the engine. ★~37k.","theCatch":"A GPU serving engine, not a laptop chat app — expect driver, CUDA/ROCm and multi-GPU ops, and you pay for the hardware. Install currently needs `--prerelease=allow`. Vs vLLM (same lane): both are open, OpenAI-compatible serving engines — benchmark your own model and traffic rather than trusting either side's charts. Vs llama.cpp: SGLang is for many concurrent users on accelerators; llama.cpp is for one machine.","verdict":"A top-tier open LLM serving engine beside vLLM — free software, the bill is GPUs; benchmark both on your workload.","addedAt":"2026-10-13","featured":false,"domain":"AI Agents","subSpecialty":"Inference","subcategory":"Inference & Local Models","tags":["self-host","permissive","active"],"capabilities":["inference","llm-serving","gpu"],"surfaces":["Self-host"],"ecosystem":["Python"],"licenseModel":"Apache-2.0","githubStars":37296,"starsPerWeek":257,"access":"free","archived":false,"repo":{"createdAt":"2024-01-08","pushedAt":"2026-10-19","license":"Apache-2.0","archived":false,"checkedAt":"2026-10-19"},"links":{"self":"/api/gems/sglang","html":"/gems/sglang","markdown":"/gems/sglang.md"}},{"slug":"llm-d","name":"llm-d","tagline":"Apache-2.0 Kubernetes stack for distributed LLM inference","category":"devops","url":"https://www.llm-d.ai","github":"https://github.com/llm-d/llm-d","pricing":"open-source","hook":"llm-d sits above model servers like vLLM and SGLang on Kubernetes, adding prefix-cache-aware routing, tiered KV-cache offload, prefill/decode disaggregation and SLO-aware autoscaling so a cluster serves real traffic efficiently.","whatsGood":"Apache-2.0 (llm-d/llm-d), release v0.10.0. A CNCF Sandbox project founded by Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA. Ships \"well-lit path\" guides with Helm charts and benchmarked recipes; most users start from the Optimized Baseline. Also covers OpenAI-compatible batch processing and wide expert-parallelism for very large models. Reproducible benchmarks are published on Prism. ★~4.8k.","theCatch":"Only makes sense if you already run Kubernetes and accelerators at scale — for one model on one GPU, a model server alone is enough. The headline gains in the README (for example 3x throughput from prefix-cache-aware routing) come from partner and vendor deployments on high-end GPUs, so measure your own. Still pre-1.0. Vs NVIDIA Dynamo (same layer): llm-d is Kubernetes-native and multi-vendor under the CNCF; Dynamo is NVIDIA-led.","verdict":"The open, Kubernetes-native layer for multi-GPU LLM serving — free, but overkill below cluster scale.","addedAt":"2026-10-13","featured":false,"domain":"AI Agents","subSpecialty":"Inference","subcategory":"Inference & Local Models","tags":["self-host","permissive","local-llm","kubernetes","active"],"capabilities":["inference","llm-serving","kubernetes"],"surfaces":["Self-host"],"ecosystem":["Multi-platform"],"licenseModel":"Apache-2.0","githubStars":4859,"starsPerWeek":63,"access":"free","archived":false,"repo":{"createdAt":"2025-04-29","pushedAt":"2026-10-19","license":"Apache-2.0","archived":false,"checkedAt":"2026-10-19"},"links":{"self":"/api/gems/llm-d","html":"/gems/llm-d","markdown":"/gems/llm-d.md"}},{"slug":"uzu","name":"uzu","tagline":"MIT on-device inference engine for Apple Silicon apps","category":"agents","url":"https://trymirai.com","github":"https://github.com/trymirai/uzu","pricing":"open-source","hook":"uzu is a Rust inference engine for running models inside your own app on Apple devices, with Swift, Python and TypeScript bindings, tool calls, structured output and an OpenAI-compatible server mode.","whatsGood":"MIT (trymirai/uzu), release 0.6.3. Native Rust crate; Swift bindings via uniffi, Python via pyo3, TypeScript via napi-rs. Metal and CPU backends, using unified memory on Apple devices; targets macOS, iOS and Mac Catalyst. One high-level API covers local chat sessions, structured output, tool calls, and optional cloud models through your own OpenAI key. A CLI can browse, download, benchmark and serve models over /v1/chat/completions. ★~2.2k.","theCatch":"Apple-first: Linux, Windows and WASM targets are still listed as in progress. It uses its own model format — you download Mirai's converted models or export one yourself with their lalamo tool, so you cannot drop in an arbitrary GGUF. The Rust quick start pulls from the git main branch. Backed by a company (Mirai). Vs Rapid-MLX (a local server for coding agents on a Mac): uzu is an embeddable engine for shipping AI inside iOS and macOS apps.","verdict":"The pick for on-device models inside an iOS or macOS app — MIT and free, but Apple-only for now and tied to its own model format.","addedAt":"2026-10-13","featured":false,"domain":"AI Agents","subSpecialty":"Local Inference","subcategory":"Inference & Local Models","tags":["sdk","self-host","local-first","permissive","local-llm","active"],"capabilities":["inference","local","sdk"],"surfaces":["SDK","Self-host"],"ecosystem":["Rust","Python","Node.js"],"licenseModel":"OSS/MIT","githubStars":2398,"starsPerWeek":35,"access":"free","archived":false,"repo":{"createdAt":"2025-06-23","pushedAt":"2026-10-18","license":"MIT","archived":false,"checkedAt":"2026-10-19"},"links":{"self":"/api/gems/uzu","html":"/gems/uzu","markdown":"/gems/uzu.md"}},{"slug":"tokenspeed","name":"TokenSpeed","tagline":"MIT LLM inference engine built for agent workloads","category":"agents","url":"https://github.com/lightseekorg/tokenspeed","github":"https://github.com/lightseekorg/tokenspeed","pricing":"open-source","hook":"TokenSpeed is an inference server tuned for agentic workloads — long multi-turn sessions, heavy prefix reuse and high per-user token speed — aiming for TensorRT-LLM-level performance with vLLM-style usability.","whatsGood":"MIT (lightseekorg/tokenspeed), release v0.1.1. Kernels are a separate, pluggable subsystem with a public API, and the project publishes day-zero support and benchmarks for new open models on NVIDIA GB300. ★~2.5k.","theCatch":"Very early (v0.1.1), so expect breaking changes. The kernel work targets recent NVIDIA data-center GPUs such as Blackwell, so gains on older or consumer cards are unclear. It covers fewer models than vLLM — check yours is supported before you switch.","verdict":"A fast-moving inference engine for agent traffic on top-end NVIDIA hardware — benchmark it beside vLLM before committing.","addedAt":"2026-10-16","featured":false,"domain":"AI Agents","subSpecialty":"Inference","subcategory":"Inference & Local Models","tags":["self-host","permissive","local-llm","active","young-repo"],"capabilities":["inference","gpu"],"surfaces":["Self-host"],"ecosystem":["Python"],"licenseModel":"OSS/MIT","githubStars":2703,"starsPerWeek":114,"access":"free","archived":false,"repo":{"createdAt":"2026-05-06","pushedAt":"2026-10-19","license":"MIT","archived":false,"checkedAt":"2026-10-19"},"links":{"self":"/api/gems/tokenspeed","html":"/gems/tokenspeed","markdown":"/gems/tokenspeed.md"}},{"slug":"comfyui","name":"ComfyUI","tagline":"GPL-3.0 node-graph engine and API for local image, video and audio models","category":"agents","url":"https://www.comfy.org","github":"https://github.com/Comfy-Org/ComfyUI","pricing":"open-source","hook":"ComfyUI runs open diffusion and media models on your own GPU through a visual node graph: you wire loaders, samplers, LoRAs and post-processing into a workflow, then run it from the UI or call it from your app over its local API.","whatsGood":"GPL-3.0 (Comfy-Org/ComfyUI), release v0.42.1 (2026-10-19). Native support covers image models such as SDXL, SD3.5, Flux.1/Flux.2 and Qwen Image, image editing, video (Wan 2.x, LTX-Video, HunyuanVideo), audio and 3D. Queueing is asynchronous, only the changed part of a graph re-runs, and VRAM/RAM management with model offloading and quantized models lets big models run on modest cards. Reusable subgraphs, templates and App Mode turn a complex graph into a simple form, and the API slots a workflow into a production pipeline. Desktop app for Windows and macOS, manual install everywhere. ★~139k.","theCatch":"It is an engine you operate, not a one-click generator: node graphs have a learning curve, and the large custom-node ecosystem is third-party Python you are running with full access to your machine, so vet what you install. Real speed needs a decent GPU. Comfy Cloud and the partner nodes for closed models are paid; the local engine is free. GPL-3.0 matters if you plan to redistribute a modified build.","verdict":"The default local engine for generative media pipelines — keep custom nodes to ones you trust.","addedAt":"2026-10-19","featured":false,"domain":"AI Agents","subSpecialty":"Agents","subcategory":"Inference & Local Models","tags":["api","web-ui","self-host","copyleft","local-llm","voice","workflows","active","fast-stars"],"capabilities":["api"],"surfaces":["API","Self-host"],"ecosystem":["Multi-platform"],"licenseModel":"GPL-3.0","githubStars":138843,"starsPerWeek":709,"access":"free","archived":false,"repo":{"createdAt":"2023-01-17","pushedAt":"2026-10-19","license":"GPL-3.0","archived":false,"checkedAt":"2026-10-19"},"links":{"self":"/api/gems/comfyui","html":"/gems/comfyui","markdown":"/gems/comfyui.md"}}],"meta":{"total":15,"count":15,"offset":0,"limit":null,"sort":"curated","sorts":["curated","catalog","new","stars","trending","name"],"filters":{"subcategory":"inference-and-local-models"},"next":null,"categories":["agents","nextjs","python","shopify","databases","devops","design","observability","rust","payments","comms","notifications","libraries"],"tags":["mcp","cli","tui","api","sdk","desktop-app","web-ui","editor-plugin","self-host","local-first","free-tier","paid-only","paid-led","permissive","copyleft","source-available","license-unclear","coding-agents","local-llm","rag","memory","evals","browser-automation","scraping","sandbox","skills","voice","security","auth","payments","realtime","postgres","vector","kubernetes","containers","observability","testing","workflows","graphics-3d","git","active","stale","archived","young-repo","fast-stars"]}}