Stack Gems
Menu
WorkbenchBestNewSkillsLogAboutSubmit
← Catalog

NVIDIA Dynamo

Apache-2.0 datacenter-scale LLM inference orchestration above vLLM, SGLang and TensorRT-LLM

AI Agents / Inference · Apache-2.0 · 8.3k

Reviewed 2026-10-06 · last push 2026-10-19 · repo checked 2026-10-19

Visit siteGitHub

What's Good

Apache-2.0 (ai-dynamo/dynamo; a few test-data files are MIT). Free path: `pip install ai-dynamo` or pull the prebuilt runtime containers from NVIDIA NGC, then start a frontend plus a vLLM or SGLang worker. KV-aware routing avoids recomputing shared prefixes, the KV Block Manager spills cache to CPU, SSD or object storage, and the Planner right-sizes prefill and decode pools against latency targets. Kubernetes path includes a Gateway API Inference Extension plugin. The 1.0 line adds per-request hints for agentic workloads, and the repo ships agent skills for deploying and debugging Dynamo. Rust core, Python extension points. ★~8.2k.

The Catch

Built for multi-GPU, multi-node serving — the README itself says one model on one GPU only needs the engine (see vLLM, or llama-swap for a local model switcher). Realistically NVIDIA hardware and Kubernetes; the headline speedups are vendor and partner benchmarks on GB200/GB300-class racks, so measure on your own GPUs. Lots of moving parts (frontend, router, workers, planner, discovery backend) and a fast release cadence: pin a version. Vs vLLM Semantic Router (model choice per request) and plain vLLM: Dynamo is the cluster scheduler, not a model router or an engine.

Verdict

Apache-2.0 cluster-level LLM serving when you run vLLM/SGLang/TRT-LLM across many GPUs — free, but overkill below multi-GPU scale and benchmarks are vendor-run.

Embed

Maintainer? Add the badge to your README. It is free, links to this review, and there is no paid version.

Reviewed on Stack Gems

Markdown

[![Reviewed on Stack Gems](https://stackgems.com/badge/nvidia-dynamo.svg)](https://stackgems.com/gems/nvidia-dynamo)

HTML

<a href="https://stackgems.com/gems/nvidia-dynamo"><img src="https://stackgems.com/badge/nvidia-dynamo.svg" alt="NVIDIA Dynamo reviewed on Stack Gems" height="20"></a>

Related

AI Agents →