{"data":{"slug":"llm-d","name":"llm-d","tagline":"Apache-2.0 Kubernetes stack for distributed LLM inference","category":"devops","categoryName":"DevOps","url":"https://www.llm-d.ai","github":"https://github.com/llm-d/llm-d","pricing":"open-source","hook":"llm-d sits above model servers like vLLM and SGLang on Kubernetes, adding prefix-cache-aware routing, tiered KV-cache offload, prefill/decode disaggregation and SLO-aware autoscaling so a cluster serves real traffic efficiently.","whatsGood":"Apache-2.0 (llm-d/llm-d), release v0.10.0. A CNCF Sandbox project founded by Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA. Ships \"well-lit path\" guides with Helm charts and benchmarked recipes; most users start from the Optimized Baseline. Also covers OpenAI-compatible batch processing and wide expert-parallelism for very large models. Reproducible benchmarks are published on Prism. ★~4.8k.","theCatch":"Only makes sense if you already run Kubernetes and accelerators at scale — for one model on one GPU, a model server alone is enough. The headline gains in the README (for example 3x throughput from prefix-cache-aware routing) come from partner and vendor deployments on high-end GPUs, so measure your own. Still pre-1.0. Vs NVIDIA Dynamo (same layer): llm-d is Kubernetes-native and multi-vendor under the CNCF; Dynamo is NVIDIA-led.","verdict":"The open, Kubernetes-native layer for multi-GPU LLM serving — free, but overkill below cluster scale.","addedAt":"2026-10-13","featured":false,"domain":"AI Agents","subSpecialty":"Inference","subcategory":"Inference & Local Models","tags":["self-host","permissive","local-llm","kubernetes","active"],"capabilities":["inference","llm-serving","kubernetes"],"surfaces":["Self-host"],"ecosystem":["Multi-platform"],"licenseModel":"Apache-2.0","githubStars":4859,"starsPerWeek":63,"access":"free","archived":false,"repo":{"createdAt":"2025-04-29","pushedAt":"2026-10-19","license":"Apache-2.0","archived":false,"checkedAt":"2026-10-19"}},"related":[{"slug":"github-actions","name":"GitHub Actions","tagline":"CI/CD that lives where your code does"},{"slug":"docker","name":"Docker","tagline":"Container runtime that changed deployment"},{"slug":"kubernetes","name":"Kubernetes","tagline":"Container orchestration at scale"},{"slug":"pulumi","name":"Pulumi","tagline":"Infrastructure as code in real languages"},{"slug":"fly-io","name":"Fly.io","tagline":"Deploy apps to VMs close to your users"}],"links":{"self":"/api/gems/llm-d","html":"/gems/llm-d","markdown":"/gems/llm-d.md","category":"/api/gems?category=devops","categoryPage":"/stacks/devops"}}