SMG (Shepherd Model Gateway)

Apache-2.0 Rust gateway that routes OpenAI and Anthropic API traffic across vLLM, SGLang, TRT-LLM and cloud models

AI Agents / Agents · Apache-2.0 · 656

Reviewed 2026-10-21 · last push 2026-10-21 · repo checked 2026-10-21

What's Good

Apache-2.0 (smg-project/smg), release v1.13.0 (2026-10-15), commit on 2026-10-21. ★~660 (~14 a week). It manages worker lifecycles, routes with KV-cache awareness, and adds chat-history storage, MCP tooling, multi-tenant auth and observability. Ships via Docker and PyPI. Useful when you run several inference engines or GPUs and want one endpoint that balances across them.

The Catch

It is the layer in front of an engine, not an engine. The SGLang gem serves the model; SMG routes traffic to SGLang, vLLM, TRT-LLM or cloud APIs. If you run one model on one box, you don't need it. The project is young (repo started November 2025) and small, so expect gaps against heavier gateways.

Verdict

A serious open gateway for multi-engine LLM fleets; overkill for a single server.

See also: SGLang

Embed

Maintainer? Add the badge to your README. It is free, links to this review, and there is no paid version.

Markdown

[![Reviewed on Stack Gems](https://stackgems.com/badge/smg.svg)](https://stackgems.com/gems/smg)

HTML

<a href="https://stackgems.com/gems/smg"><img src="https://stackgems.com/badge/smg.svg" alt="SMG (Shepherd Model Gateway) reviewed on Stack Gems" height="20"></a>

AI Agents →

Related