Tag
Evals & tracing tools
Evaluation, tracing, LLM observability. 22 reviewed tools: 20 free or open source, 2 paid or paid-led.
Order: free and open source first, then hosted products with a real free tier, then paid-led; catalog order inside each group, with quiet or archived repos last. Not a score, and no tool pays for its place.
22 tools
- LangfuseOpen-source LLM observability you can self-host
Free / OSS · Open-Core · 36.1k GitHub stars
VerdictThe LLM observability platform for teams who want to own their data.
CatchLicense: the core is MIT, but anything under the ee/ directories is under a separate enterprise license — check which features you rely on before calling your deployment fully MIT. Self-hosting means self-operating—you are running Postgres, ClickHouse, Redis, and the app. Observability-first means evals are layered on, not central.
- MastraTypeScript agent runtime with typed agents and workflows
Free / OSS · OSS/MIT
VerdictTypeScript agent framework for teams building agent systems.
CatchFramework, not a coding CLI. Mastra helps you build agents; it is not an agent itself. Learning curve for the runtime concepts. You are committing to their abstractions.
- PromptfooMIT LLM evals & red-teaming CLI; Community free, 10k probes/mo
Free / OSS · Freemium · 26.2k GitHub stars
VerdictMIT eval/red-team CLI. Free core; probe cap + Enterprise sales wall.
CatchVs DeepEval/Braintrust: YAML/CLI-first, not pytest-native Python or Braintrust SaaS. Community caps red-team probes (~10k/mo); Enterprise/On-Prem are contact-sales (SSO, dashboards, dedicated runner). LLM API costs are yours. Acquisition governance is a multi-year bet.
- DeepEvalApache-2.0 pytest-style LLM/agent evals; Confident AI is the paid cloud
Free / OSS · Apache-2.0 · 19k GitHub stars
VerdictApache-2.0 LLM unit tests. Free locally; Confident AI starts free then $200.
CatchOSS is the runner — collaboration/dashboards/monitoring are Confident AI (Free: 2 seats/1 project/5 runs/wk; Starter $200/mo; Team $2k/mo). LLM-as-judge burns provider tokens. Vs Ragas: broader agent/chat coverage, not RAG-only.
- Google ADKApache-2.0 code-first agent build/eval/deploy kit (Google)
Free / OSS · Apache-2.0 · 21.9k GitHub stars
VerdictApache-2.0 Google agent kit. Code-first build/eval/deploy; Vertex optional, not mandatory.
CatchVs Gemini CLI / Antigravity CLI (terminal coding agents), CrewAI/LangGraph/Agno (orchestration frameworks), and Daytona/E2B (sandboxes): ADK is the Google build/eval/deploy kit, not a coding CLI and not a sandbox fleet. Vertex/GCP gravity is real for enterprise deploy; OSS still runs without a Google Cloud bill if you stay local/BYO.
- OpenAI Agents SDKMIT multi-agent workflows — tools, handoffs, tracing, MCP
Free / OSS · OSS/MIT · 30.2k GitHub stars
VerdictMIT OpenAI agent workflow SDK. Build multi-agent loops; API spend is separate.
CatchVs openai-codex-cli (terminal coding agent), Vercel AI SDK (app AI primitives), and Microsoft Agent Framework: this is the workflow SDK, not the Codex CLI seat. OpenAI API keys and hosted tools meter; sandbox/realtime extras add surface area. Do not confuse with ChatGPT product seats.
- MCPJamOSS MCP testing, debugging, and evals platform
Free / OSS · Apache-2.0 · 2.3k GitHub stars
VerdictApache-2.0 MCP test platform. Use for real-client MCP debugging/evals; watch the proprietary evals carve-out.
CatchVs MCP Inspector (official MIT debugger): MCPJam is the broader test/eval platform with hosted gravity, not only the reference inspector. Some `/server/services` and evals paths are separately licensed proprietary portions — treat the free OSS inspector path as the bar, not every hosted feature.
- AX CheckHosted free agent-experience checklist + recorded coding-agent sessions
Free / OSS · OSS/MIT
VerdictFree hosted AX smoke test. Grades how agents use your product, not how good your model is.
CatchGrades are provisional and technical-only — treat the report as a useful first pass, not a product-wide quality certificate. Vs Promptfoo / DeepEval: AX Check grades product AX (docs, CLI, MCP, SDK), not model quality, eval scores, or safety behavior. Hosted free is convenient; verify the session evidence and rerun after interface changes.
- tRPC-Agent-GoApache-2.0 Go production agent framework — GraphAgent, MCP, A2A
Free / OSS · Apache-2.0 · 1.9k GitHub stars
VerdictApache-2.0 Go agent framework when GraphAgent + MCP/A2A belong inside Go services — skip if your team is Python-first and does not want another runtime.
CatchGo-native stack with tRPC-ecosystem gravity — not a drop-in for Python LangGraph/Crew shops. Not a hosted product: you build and run the service. Pick it when Go is already the runtime, not as a reason to leave a working Python agent stack.
- Arize PhoenixELv2 self-host AI observability & eval — traces, datasets, playground
Free / OSS · Open-Core · 11.9k GitHub stars
VerdictELv2 self-host AI obs/eval when Phoenix’s tracing+evals fit and you accept source-available — not Langfuse MIT, not Braintrust CI-eval.
CatchNOT OSI open source — ELv2 forbids offering Phoenix as a competing hosted SaaS. Vs Langfuse (MIT full-stack self-host) and Braintrust (eval-first freemium): Phoenix is Arize’s ELv2 observability/eval kit with strong notebook/embedding UX, not MIT and not Braintrust’s CI-eval product. Arize cloud is the upsell when you want managed multi-tenant.
- iFixAiApache-2.0 agents eval/audit — pip install, premium ≠ paywall
Free / OSS · Apache-2.0 · 30.8k GitHub stars
VerdictApache agents eval/audit when you want pip-installable scoring without an eval SaaS gate — not Braintrust/Langfuse/Ragas/Promptfoo’s lanes; premium ≠ required.
CatchVs Braintrust / Langfuse / Ragas / Promptfoo: Braintrust is eval-first commercial; Langfuse is MIT obs/self-host; Ragas is Apache RAG/agent metrics + CI scorers; Promptfoo is OSS prompt/agent eval — iFixAi’s catch is agents eval/audit as an installable audit kit, not Braintrust seats, not Langfuse traces, not Ragas metrics alone, not Promptfoo prompt suites. Stars grow fast (~47 a week): judge the audit loop, not the counter.
- Monty (Pydantic)MIT in-process Rust Python sandbox — OSS Monty only
Free / OSS · OSS/MIT · 8.7k GitHub stars
VerdictMIT in-process Python sandbox when agents need Pydantic Monty OSS — not E2B/AIO/CubeSandbox cloud or KVM seats; ignore the ★ spike.
Catch★8499 in about 41 weeks (~207 a week): judge it on the sandbox shape, not the star count. Vs E2B / AIO Sandbox / CubeSandbox: those are hosted Firecracker seats, broader sandbox/browser/MCP runtimes, or self-hosted RustVMM/KVM density — Monty’s catch is in-process Rust Python sandbox from Pydantic, not a sandbox SaaS seat. Do not confuse OSS Monty packages with Full Monty commercial; pin the MIT crates/npm/PyPI names.
- JaegerApache-2.0 CNCF distributed tracing, OTLP-native
Free / OSS · Apache-2.0 · 23.3k GitHub stars
VerdictThe default self-hosted trace backend for OpenTelemetry — start with all-in-one, then budget for the storage.
CatchTraces only — metrics and logs live elsewhere, so most teams pair it with Prometheus/Grafana or pick an all-in-one such as SigNoz. In-memory storage is for local use; production means running and sizing a storage backend yourself. v2 configuration differs from v1 CLI flags, so older guides may not match.
- OpikApache-2.0 LLM and agent tracing, evaluation and monitoring by Comet
Free / OSS · Apache-2.0 · 22.8k GitHub stars
VerdictA strong self-hostable Langfuse alternative with evals built in — choose it if CI evaluation matters most to you.
CatchSelf-hosting is a multi-service stack, not a single binary, so budget ops time or use Opik Cloud, Comet's hosted version with its own free tier and paid plans. README and docs links lean toward Comet sign-up. It overlaps heavily with Langfuse and Phoenix, so pick one tracing backend per project.
- LangWatchApache-2.0 LLM tracing, agent tests, evals, gateway
Free / OSS · Apache-2.0 · 5k GitHub stars
VerdictWorth a look if you want agent tests and an LLM gateway in the same tool as your traces.
CatchIt tries to be four products at once, so expect a broad surface that overlaps with Langfuse, Opik and LiteLLM. The production self-host is more than a single Node process. The managed LangWatch Cloud is the commercial path, and README links lean that way.
- OdigosApache-2.0 distributed tracing without code changes
Free / OSS · Apache-2.0 · 3.7k GitHub stars
VerdictThe quickest way to get traces from a fleet you cannot re-instrument.
CatchAuto-instrumentation shows you request flow but not your business attributes; you will still add manual spans for that. Some advanced eBPF instrumentation is in the commercial Odigos Enterprise. It is another operator to run in the cluster.
- LaminarApache-2.0 tracing, evals and signals for AI agents
Free / OSS · Apache-2.0 · 3.4k GitHub stars
VerdictStrongest pick when you trace browser and computer-use agents.
CatchSelf-hosting is a multi-service stack. Signals run an LLM over your agent runs, which adds cost and sends trace content to a model. The hosted Laminar cloud is the main product path. Crowded space next to Langfuse, Opik and LangWatch.
- OpenLITApache-2.0 OTel tracing, evals and guardrails for agents
Free / OSS · Apache-2.0 · 2.9k GitHub stars
VerdictA solid self-hosted LLM observability option, especially if you also watch GPU cost.
CatchThe README is heavy on the new agent-harness-engineering framing; underneath it is an OTel tracing and eval platform in a crowded field. Self-hosting means running its UI and ClickHouse. Many repos and releases (SDKs, collectors) to keep in sync.
- Inspect AIMIT LLM and agent evaluation framework from the UK AI Security Institute
Free / OSS · OSS/MIT · 3.1k GitHub stars
VerdictA serious, actively developed eval framework; pin it.
CatchIt is still 0.x and releases near-daily, so pin a version or your eval results can shift under you.
- RagasApache-2.0 RAG/agent eval metrics — synthetic data + CI scorers
Free / OSS · Apache-2.0 · 16.1k GitHub stars · quiet 180d+
VerdictApache-2.0 RAG/agent eval metrics when faithfulness and context precision beat a hosted eval bill — not DeepEval pytest, not Promptfoo red-team.
CatchVs DeepEval (pytest-native agent/chat metrics + Confident AI cloud) and Promptfoo (YAML/CLI red-team): Ragas is RAG-first metric library + synthetic data, not a full red-team harness and not Confident AI dashboards. LLM-as-judge burns provider tokens; cloud extras (if any) are optional — core library stays OSS.
- BraintrustEval-first LLM platform that blocks bad prompts in CI
Paid-led · Freemium
VerdictThe platform for teams whose deployment loop is change-measure-ship.
CatchEval-first means observability is secondary—if your primary loop is debugging traces, look elsewhere. Learning curve to build eval datasets that actually represent production.
- LangSmithLangChain freemium obs/eval/deploy — free Developer 5k traces/mo
Paid-led · Freemium · 15 GitHub stars
VerdictFreemium LangChain obs/eval when the LangGraph stack wants one vendor pane — not Langfuse MIT self-host; free until ~5k traces.
CatchThe product is hosted freemium (self-host exists on higher plans) — not a MIT self-host default like Langfuse. Vs Langfuse / Braintrust / Arize Phoenix: LangSmith is the LangChain-native commercial control plane; Langfuse is OSS self-host; Braintrust is eval-first; Phoenix is ELv2 Arize. Free tier is real but thin for busy agent fleets — budget traces early.
FAQ
- What is the top free pick among Evals & tracing tools?
- Langfuse leads the Stack Gems list for Evals & tracing tools. Verdict: The LLM observability platform for teams who want to own their data.
- How many Evals & tracing tools does Stack Gems review?
- 22: 20 free or open source, 2 paid or paid-led. Each review has a verdict and an honest Catch; no tool pays for its spot.
- Which Evals & tracing tools can I self-host for free?
- Langfuse, Mastra, DeepEval, Google ADK, OpenAI Agents SDK, MCPJam, tRPC-Agent-Go, Arize Phoenix and 10 more.
- Which Evals & tracing tools have the most GitHub stars?
- Langfuse (36.1k), iFixAi (30.8k), OpenAI Agents SDK (30.2k), per GitHub on 2026-10-21. Stars measure attention, not fit; read the Catch.
- What is the catch with Langfuse?
- License: the core is MIT, but anything under the ee/ directories is under a separate enterprise license — check which features you rely on before calling your deployment fully MIT.