llm-d

Apache-2.0 Kubernetes stack for distributed LLM inference

AI Agents / Inference · Apache-2.0 · 4.9k

Reviewed 2026-10-13 · last push 2026-10-19 · repo checked 2026-10-19

What's Good

Apache-2.0 (llm-d/llm-d), release v0.10.0. A CNCF Sandbox project founded by Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA. Ships "well-lit path" guides with Helm charts and benchmarked recipes; most users start from the Optimized Baseline. Also covers OpenAI-compatible batch processing and wide expert-parallelism for very large models. Reproducible benchmarks are published on Prism. ★~4.8k.

The Catch

Only makes sense if you already run Kubernetes and accelerators at scale — for one model on one GPU, a model server alone is enough. The headline gains in the README (for example 3x throughput from prefix-cache-aware routing) come from partner and vendor deployments on high-end GPUs, so measure your own. Still pre-1.0. Vs NVIDIA Dynamo (same layer): llm-d is Kubernetes-native and multi-vendor under the CNCF; Dynamo is NVIDIA-led.

Verdict

The open, Kubernetes-native layer for multi-GPU LLM serving — free, but overkill below cluster scale.

Embed

Maintainer? Add the badge to your README. It is free, links to this review, and there is no paid version.

Markdown

[![Reviewed on Stack Gems](https://stackgems.com/badge/llm-d.svg)](https://stackgems.com/gems/llm-d)

HTML

<a href="https://stackgems.com/gems/llm-d"><img src="https://stackgems.com/badge/llm-d.svg" alt="llm-d reviewed on Stack Gems" height="20"></a>

DevOps →

Related