vLLM Semantic Router

Apache-2.0 Mixture-of-Models router — pick or compose the model path per request

AI Agents / Agents · Apache-2.0 · 6.2k

Reviewed 2026-10-05 · last push 2026-10-19 · repo checked 2026-10-19

What's Good

Apache-2.0 (vllm-project/semantic-router, v0.4.0; PyPI `vllm-sr` 0.4.0). Go core in the vLLM org. Stable entrypoints resolve to “recipes” that can attach retrieval, memory, tool filtering, caching, safety checks, and verification. Works with Envoy and existing inference runtimes rather than replacing them; Docker, Kubernetes, and hardware paths documented; public playground. ★~6k.

The Catch

Platform, not a one-file proxy: it coordinates alongside Envoy and your model servers, so budget real ops time; the docs are written for heterogeneous fleets (cloud/data center/edge) more than a two-model hobby setup. Install is a `curl | bash` script (pip/uv paths exist — prefer those if you audit installs). Vs litellm / bifrost (multi-provider gateways) and weave-router (ELv2 embedder router): this is Apache policy-driven semantic routing that composes model paths.

Verdict

Apache MoM router when one endpoint must route across many models and hardware by intent and cost — overkill if LiteLLM fallbacks already cover you.

Embed

Maintainer? Add the badge to your README. It is free, links to this review, and there is no paid version.

Markdown

[![Reviewed on Stack Gems](https://stackgems.com/badge/vllm-semantic-router.svg)](https://stackgems.com/gems/vllm-semantic-router)

HTML

<a href="https://stackgems.com/gems/vllm-semantic-router"><img src="https://stackgems.com/badge/vllm-semantic-router.svg" alt="vLLM Semantic Router reviewed on Stack Gems" height="20"></a>

AI Agents →

Related