Discover
Agent eval frameworks
Prompt and agent evaluation harnesses: CI red-teams, pytest evals, and observability that is not a scoreboard.
4 tools · Stars / Runtime / License chips · the Catch, not a ranking
- PromptfooMIT LLM evals & red-teaming CLI; Community free, 10k probes/mo★ 26.1kNode.js · Multi-platformFreemium
Vs DeepEval/Braintrust: YAML/CLI-first, not pytest-native Python or Braintrust SaaS. Community caps red-team probes (~10k/mo); Enterprise/On-Prem are contact-sales (SSO, dashboards, dedicated runner). LLM API costs are yours. Acquisition governance is a multi-year bet.
- DeepEvalApache-2.0 pytest-style LLM/agent evals; Confident AI is the paid cloud★ 18.9kPythonApache-2.0
OSS is the runner — collaboration/dashboards/monitoring are Confident AI (Free: 2 seats/1 project/5 runs/wk; Starter $200/mo; Team $2k/mo). LLM-as-judge burns provider tokens. Vs Ragas: broader agent/chat coverage, not RAG-only.
- BraintrustEval-first LLM platform that blocks bad prompts in CIMulti-platformFreemium
Eval-first means observability is secondary—if your primary loop is debugging traces, look elsewhere. Learning curve to build eval datasets that actually represent production.
- LangfuseOpen-source LLM observability you can self-host★ 36kMulti-platformOpen-Core
License: the core is MIT, but anything under the ee/ directories is under a separate enterprise license — check which features you rely on before calling your deployment fully MIT. Self-hosting means self-operating—you are running Postgres, ClickHouse, Redis, and the app. Observability-first means evals are layered on, not central.