# NVIDIA Dynamo

> Apache-2.0 datacenter-scale LLM inference orchestration above vLLM, SGLang and TensorRT-LLM

**Category:** AI Agents  
**Pricing:** open-source  
**URL:** https://docs.nvidia.com/dynamo/  
**GitHub:** https://github.com/ai-dynamo/dynamo  
**Added:** 2026-10-06

---

## The Hook

Dynamo is the orchestration layer above inference engines: it keeps vLLM, SGLang or TensorRT-LLM as the engine and turns a cluster of GPUs into one coordinated, OpenAI-compatible serving system with disaggregated prefill/decode, KV-aware routing, KV cache offload and SLA-driven autoscaling.

## What's Good

Apache-2.0 (ai-dynamo/dynamo; a few test-data files are MIT). Free path: `pip install ai-dynamo` or pull the prebuilt runtime containers from NVIDIA NGC, then start a frontend plus a vLLM or SGLang worker. KV-aware routing avoids recomputing shared prefixes, the KV Block Manager spills cache to CPU, SSD or object storage, and the Planner right-sizes prefill and decode pools against latency targets. Kubernetes path includes a Gateway API Inference Extension plugin. The 1.0 line adds per-request hints for agentic workloads, and the repo ships agent skills for deploying and debugging Dynamo. Rust core, Python extension points. ★~8.2k.

## The Catch

Built for multi-GPU, multi-node serving — the README itself says one model on one GPU only needs the engine (see vLLM, or llama-swap for a local model switcher). Realistically NVIDIA hardware and Kubernetes; the headline speedups are vendor and partner benchmarks on GB200/GB300-class racks, so measure on your own GPUs. Lots of moving parts (frontend, router, workers, planner, discovery backend) and a fast release cadence: pin a version. Vs vLLM Semantic Router (model choice per request) and plain vLLM: Dynamo is the cluster scheduler, not a model router or an engine.

## Verdict

Apache-2.0 cluster-level LLM serving when you run vLLM/SGLang/TRT-LLM across many GPUs — free, but overkill below multi-GPU scale and benchmarks are vendor-run.