What is the top free pick among LLM eval and tracing tools?

Arize Phoenix leads the Stack Gems list for LLM eval and tracing tools. Verdict: ELv2 self-host AI obs/eval when Phoenix’s tracing+evals fit and you accept source-available — not Langfuse MIT, not Braintrust CI-eval.

How many LLM eval and tracing tools does Stack Gems review?

9: 8 free or open source, 1 paid or paid-led. Each review has a verdict and an honest Catch; no tool pays for its spot.

Which LLM eval and tracing tools can I self-host for free?

Arize Phoenix, e2e (TesterArmy), Opik, LangWatch, HolmesGPT, Laminar, OpenLIT and Ragas.

Which LLM eval and tracing tools have the most GitHub stars?

Opik (22.7k), Ragas (16.1k), Arize Phoenix (11.9k), per GitHub on 2026-10-19. Stars measure attention, not fit; read the Catch.

What is the catch with Arize Phoenix?

NOT OSI open source — ELv2 forbids offering Phoenix as a competing hosted SaaS.

Best of · AI Agents

Best LLM eval and tracing tools (2026)

Stack Gems reviews 9 LLM eval and tracing tools: 8 free or open source, 1 paid or paid-led. These are the top 9, each with its verdict and the Catch we would want to know before installing it.

Order: free and open source first, then hosted products with a real free tier, then paid-led; catalog order inside each group, with quiet or archived repos last. Not a score, and no tool pays for its place.

9 reviewed · 9 shown in full

1.Arize PhoenixELv2 self-host AI observability & eval — traces, datasets, playground
Free / OSS · Open-Core · 11.9k GitHub stars
VerdictELv2 self-host AI obs/eval when Phoenix’s tracing+evals fit and you accept source-available — not Langfuse MIT, not Braintrust CI-eval.
CatchNOT OSI open source — ELv2 forbids offering Phoenix as a competing hosted SaaS. Vs Langfuse (MIT full-stack self-host) and Braintrust (eval-first freemium): Phoenix is Arize’s ELv2 observability/eval kit with strong notebook/embedding UX, not MIT and not Braintrust’s CI-eval product. Arize cloud is the upsell when you want managed multi-tenant.
2.e2e (TesterArmy)Apache-2.0 agent-driven e2e tests for web and mobile — replays without model calls
Free / OSS · Apache-2.0 · 11.3k GitHub stars
VerdictApache agentic e2e when flaky selectors cost more than tokens — record-and-replay keeps CI cheap; pin minors and turn telemetry off.
CatchTelemetry: the CLI sends anonymous usage data by default — `npx e2e telemetry disable` or `E2E_TELEMETRY_DISABLED=1`. Pre-1.0: APIs and config can change between minors. Built by TesterArmy, a commercial agentic-testing platform; Kernel hosted browsers and EAS simulators are optional paid add-on engines. Vs stagehand (browser automation SDK) and playwright (deterministic runner): e2e is a test framework that mixes agent steps with assertions and caches them.
3.OpikApache-2.0 LLM and agent tracing, evaluation and monitoring by Comet
Free / OSS · Apache-2.0 · 22.7k GitHub stars
VerdictA strong self-hostable Langfuse alternative with evals built in — choose it if CI evaluation matters most to you.
CatchSelf-hosting is a multi-service stack, not a single binary, so budget ops time or use Opik Cloud, Comet's hosted version with its own free tier and paid plans. README and docs links lean toward Comet sign-up. It overlaps heavily with Langfuse and Phoenix, so pick one tracing backend per project.
4.LangWatchApache-2.0 LLM tracing, agent tests, evals, gateway
Free / OSS · Apache-2.0 · 5k GitHub stars
VerdictWorth a look if you want agent tests and an LLM gateway in the same tool as your traces.
CatchIt tries to be four products at once, so expect a broad surface that overlaps with Langfuse, Opik and LiteLLM. The production self-host is more than a single Node process. The managed LangWatch Cloud is the commercial path, and README links lean that way.
5.HolmesGPTApache-2.0 CNCF AI agent for production incidents
Free / OSS · Apache-2.0 · 3.6k GitHub stars
VerdictThe most credible open SRE agent — start read-only and keep a human in the loop.
CatchIt needs read access to your infrastructure and observability data and sends that context to the LLM you configure, so pick the provider and scope permissions with care. Root-cause suggestions can be wrong and need human review, and letting it open PRs is a step to take deliberately. LLM costs scale with investigation volume.
6.LaminarApache-2.0 tracing, evals and signals for AI agents
Free / OSS · Apache-2.0 · 3.4k GitHub stars
VerdictStrongest pick when you trace browser and computer-use agents.
CatchSelf-hosting is a multi-service stack. Signals run an LLM over your agent runs, which adds cost and sends trace content to a model. The hosted Laminar cloud is the main product path. Crowded space next to Langfuse, Opik and LangWatch.
7.OpenLITApache-2.0 OTel tracing, evals and guardrails for agents
Free / OSS · Apache-2.0 · 2.9k GitHub stars
VerdictA solid self-hosted LLM observability option, especially if you also watch GPU cost.
CatchThe README is heavy on the new agent-harness-engineering framing; underneath it is an OTel tracing and eval platform in a crowded field. Self-hosting means running its UI and ClickHouse. Many repos and releases (SDKs, collectors) to keep in sync.
8.RagasApache-2.0 RAG/agent eval metrics — synthetic data + CI scorers
Free / OSS · Apache-2.0 · 16.1k GitHub stars · quiet 180d+
VerdictApache-2.0 RAG/agent eval metrics when faithfulness and context precision beat a hosted eval bill — not DeepEval pytest, not Promptfoo red-team.
CatchVs DeepEval (pytest-native agent/chat metrics + Confident AI cloud) and Promptfoo (YAML/CLI red-team): Ragas is RAG-first metric library + synthetic data, not a full red-team harness and not Confident AI dashboards. LLM-as-judge burns provider tokens; cloud extras (if any) are optional — core library stays OSS.
9.LangSmithLangChain freemium obs/eval/deploy — free Developer 5k traces/mo
Paid-led · Freemium · 13 GitHub stars
VerdictFreemium LangChain obs/eval when the LangGraph stack wants one vendor pane — not Langfuse MIT self-host; free until ~5k traces.
CatchThe product is hosted freemium (self-host exists on higher plans) — not a MIT self-host default like Langfuse. Vs Langfuse / Braintrust / Arize Phoenix: LangSmith is the LangChain-native commercial control plane; Langfuse is OSS self-host; Braintrust is eval-first; Phoenix is ELv2 Arize. Free tier is real but thin for busy agent fleets — budget traces early.

FAQ

What is the top free pick among LLM eval and tracing tools?: Arize Phoenix leads the Stack Gems list for LLM eval and tracing tools. Verdict: ELv2 self-host AI obs/eval when Phoenix’s tracing+evals fit and you accept source-available — not Langfuse MIT, not Braintrust CI-eval.
How many LLM eval and tracing tools does Stack Gems review?: 9: 8 free or open source, 1 paid or paid-led. Each review has a verdict and an honest Catch; no tool pays for its spot.
Which LLM eval and tracing tools can I self-host for free?: Arize Phoenix, e2e (TesterArmy), Opik, LangWatch, HolmesGPT, Laminar, OpenLIT and Ragas.
Which LLM eval and tracing tools have the most GitHub stars?: Opik (22.7k), Ragas (16.1k), Arize Phoenix (11.9k), per GitHub on 2026-10-19. Stars measure attention, not fit; read the Catch.
What is the catch with Arize Phoenix?: NOT OSI open source — ELv2 forbids offering Phoenix as a competing hosted SaaS.

Alternatives

Tags in this list

Related lists

JSON: /api/gems?subcategory=evals-and-tracing · MCP: best_for on /api/mcp