# llm-d

> Apache-2.0 Kubernetes stack for distributed LLM inference

**Category:** DevOps  
**Pricing:** open-source  
**URL:** https://www.llm-d.ai  
**GitHub:** https://github.com/llm-d/llm-d  
**Added:** 2026-10-13

---

## The Hook

llm-d sits above model servers like vLLM and SGLang on Kubernetes, adding prefix-cache-aware routing, tiered KV-cache offload, prefill/decode disaggregation and SLO-aware autoscaling so a cluster serves real traffic efficiently.

## What's Good

Apache-2.0 (llm-d/llm-d), release v0.10.0. A CNCF Sandbox project founded by Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA. Ships "well-lit path" guides with Helm charts and benchmarked recipes; most users start from the Optimized Baseline. Also covers OpenAI-compatible batch processing and wide expert-parallelism for very large models. Reproducible benchmarks are published on Prism. ★~4.8k.

## The Catch

Only makes sense if you already run Kubernetes and accelerators at scale — for one model on one GPU, a model server alone is enough. The headline gains in the README (for example 3x throughput from prefix-cache-aware routing) come from partner and vendor deployments on high-end GPUs, so measure your own. Still pre-1.0. Vs NVIDIA Dynamo (same layer): llm-d is Kubernetes-native and multi-vendor under the CNCF; Dynamo is NVIDIA-led.

## Verdict

The open, Kubernetes-native layer for multi-GPU LLM serving — free, but overkill below cluster scale.