{"data":[{"slug":"ragas","name":"Ragas","tagline":"Apache-2.0 RAG/agent eval metrics — synthetic data + CI scorers","category":"agents","url":"https://www.ragas.io","github":"https://github.com/vibrantlabsai/ragas","pricing":"open-source","hook":"Ragas is the Apache-2.0 eval kit for RAG and agent pipelines — faithfulness, context precision, and synthetic test sets without renting an eval SaaS.","whatsGood":"Apache-2.0 (explodinggradients/ragas → vibrantlabsai). Python metrics for RAG (faithfulness, answer relevancy, context precision/recall) plus agent/trajectory scorers and synthetic test-data generation. Runs locally or in CI against your own LLM providers. Docs at ragas.io / docs.ragas.io. ★15,869 on GitHub 2026-09-29.","theCatch":"Vs DeepEval (pytest-native agent/chat metrics + Confident AI cloud) and Promptfoo (YAML/CLI red-team): Ragas is RAG-first metric library + synthetic data, not a full red-team harness and not Confident AI dashboards. LLM-as-judge burns provider tokens; cloud extras (if any) are optional — core library stays OSS.","verdict":"Apache-2.0 RAG/agent eval metrics when faithfulness and context precision beat a hosted eval bill — not DeepEval pytest, not Promptfoo red-team.","addedAt":"2026-09-29","featured":false,"domain":"AI Agents","subSpecialty":"Agents","subcategory":"Evals & Tracing","tags":["self-host","local-first","permissive","rag","evals","observability","testing","stale"],"capabilities":["ci"],"surfaces":["Self-host","Hosted"],"ecosystem":["Multi-platform"],"licenseModel":"Apache-2.0","githubStars":16067,"starsPerWeek":89,"access":"free","archived":false,"repo":{"createdAt":"2023-05-08","pushedAt":"2026-02-24","license":"Apache-2.0","archived":false,"checkedAt":"2026-10-19"},"links":{"self":"/api/gems/ragas","html":"/gems/ragas","markdown":"/gems/ragas.md"}},{"slug":"arize-phoenix","name":"Arize Phoenix","tagline":"ELv2 self-host AI observability & eval — traces, datasets, playground","category":"agents","url":"https://phoenix.arize.com","github":"https://github.com/Arize-ai/phoenix","pricing":"source-available","hook":"Phoenix is Arize’s self-host AI observability and evaluation stack — traces, embeddings, datasets, and evals when Langfuse is the MIT default and you want Phoenix’s notebook-friendly path.","whatsGood":"Elastic License 2.0 (Arize-ai/phoenix). Open inference tracing, embeddings viz, datasets, experiments, and eval helpers; pip/Docker self-host; pairs with Arize AX cloud if you outgrow local. Docs at arize.com/docs/phoenix. ★11,645 on GitHub 2026-09-29.","theCatch":"NOT OSI open source — ELv2 forbids offering Phoenix as a competing hosted SaaS. Vs Langfuse (MIT full-stack self-host) and Braintrust (eval-first freemium): Phoenix is Arize’s ELv2 observability/eval kit with strong notebook/embedding UX, not MIT and not Braintrust’s CI-eval product. Arize cloud is the upsell when you want managed multi-tenant.","verdict":"ELv2 self-host AI obs/eval when Phoenix’s tracing+evals fit and you accept source-available — not Langfuse MIT, not Braintrust CI-eval.","addedAt":"2026-09-29","featured":false,"domain":"AI Agents","subSpecialty":"Agents","subcategory":"Evals & Tracing","tags":["self-host","source-available","license-unclear","rag","evals","observability","active"],"capabilities":[],"surfaces":["Self-host"],"ecosystem":["Multi-platform"],"licenseModel":"Open-Core","githubStars":11905,"starsPerWeek":58,"access":"free","archived":false,"repo":{"createdAt":"2022-11-09","pushedAt":"2026-10-18","license":"NOASSERTION","archived":false,"checkedAt":"2026-10-19"},"links":{"self":"/api/gems/arize-phoenix","html":"/gems/arize-phoenix","markdown":"/gems/arize-phoenix.md"}},{"slug":"testerarmy-e2e","name":"e2e (TesterArmy)","tagline":"Apache-2.0 agent-driven e2e tests for web and mobile — replays without model calls","category":"agents","url":"https://tester.army/e2e","github":"https://github.com/tester-army/e2e","pricing":"open-source","hook":"e2e lets a test say `agent.act(\"upgrade the workspace to Pro\")` and `agent.assert(...)` next to normal locators — an agent drives the app the first time, the verified steps are recorded, and later runs replay them with no model calls until the UI changes.","whatsGood":"Apache-2.0 (tester-army/e2e; npm `e2e` 0.17.0). `npx e2e init` picks web or mobile and a model provider (bring your own subscription, API key, or local model). Web engine via Playwright (Chromium/Firefox/WebKit); iOS/Android via agent-device simulators; GitHub PR-comment reporter. Tests without agent steps need no model. Docs ship inside the package (`node_modules/e2e/docs`) for coding agents. ★~3.4k.","theCatch":"Telemetry: the CLI sends anonymous usage data by default — `npx e2e telemetry disable` or `E2E_TELEMETRY_DISABLED=1`. Pre-1.0: APIs and config can change between minors. Built by TesterArmy, a commercial agentic-testing platform; Kernel hosted browsers and EAS simulators are optional paid add-on engines. Vs stagehand (browser automation SDK) and playwright (deterministic runner): e2e is a test framework that mixes agent steps with assertions and caches them.","verdict":"Apache agentic e2e when flaky selectors cost more than tokens — record-and-replay keeps CI cheap; pin minors and turn telemetry off.","addedAt":"2026-10-05","featured":false,"domain":"AI Agents","subSpecialty":"Agents","subcategory":"Evals & Tracing","tags":["self-host","permissive","testing","active","young-repo","fast-stars"],"capabilities":[],"surfaces":["Self-host"],"ecosystem":["Multi-platform"],"licenseModel":"Apache-2.0","githubStars":11258,"starsPerWeek":880,"access":"free","archived":false,"repo":{"createdAt":"2026-07-22","pushedAt":"2026-10-18","license":"Apache-2.0","archived":false,"checkedAt":"2026-10-19"},"links":{"self":"/api/gems/testerarmy-e2e","html":"/gems/testerarmy-e2e","markdown":"/gems/testerarmy-e2e.md"}},{"slug":"opik","name":"Opik","tagline":"Apache-2.0 LLM and agent tracing, evaluation and monitoring by Comet","category":"agents","url":"https://www.comet.com/docs/opik/","github":"https://github.com/comet-ml/opik","pricing":"open-source","hook":"Opik is an open-source platform for tracing, evaluating and monitoring LLM apps and agents: wrap a function with @track and you get full trace trees of LLM and tool calls, then datasets, experiments and LLM-as-a-judge metrics on top.","whatsGood":"Apache-2.0 (comet-ml/opik), release 2.2.103 (2026-10-16). The full platform is free to self-host, backend included, via ./opik.sh with Docker Compose for local use or Kubernetes for scale. It covers trace annotation, a prompt playground, hallucination and RAG metrics, online evaluation rules on production traffic, a PyTest integration for evals in CI, and an MCP server. ★~22.7k.","theCatch":"Self-hosting is a multi-service stack, not a single binary, so budget ops time or use Opik Cloud, Comet's hosted version with its own free tier and paid plans. README and docs links lean toward Comet sign-up. It overlaps heavily with Langfuse and Phoenix, so pick one tracing backend per project.","verdict":"A strong self-hostable Langfuse alternative with evals built in — choose it if CI evaluation matters most to you.","addedAt":"2026-10-18","featured":false,"domain":"AI Agents","subSpecialty":"Agents","subcategory":"Evals & Tracing","tags":["self-host","permissive","evals","observability","active"],"capabilities":[],"surfaces":["Self-host"],"ecosystem":["Multi-platform"],"licenseModel":"Apache-2.0","githubStars":22728,"starsPerWeek":126,"access":"free","archived":false,"repo":{"createdAt":"2023-05-10","pushedAt":"2026-10-19","license":"Apache-2.0","archived":false,"checkedAt":"2026-10-19"},"links":{"self":"/api/gems/opik","html":"/gems/opik","markdown":"/gems/opik.md"}},{"slug":"langwatch","name":"LangWatch","tagline":"Apache-2.0 LLM tracing, agent tests, evals, gateway","category":"agents","url":"https://langwatch.ai","github":"https://github.com/langwatch/langwatch","pricing":"open-source","hook":"LangWatch is an open-source platform to trace, test, route and govern LLM calls — from the agents you build to the coding assistants your engineers use.","whatsGood":"Apache-2.0 (langwatch/langwatch), release langwatch-3.20.1 (2026-10-02). It covers observability, simulation tests for agents and voice agents, evaluations and prompt management, plus coding-agent analytics (sessions, cost per pull request and per team), an OpenAI- and Anthropic-compatible gateway with virtual keys and budgets, and governance views. Self-hosting needs only Node.js for a quick start, with production docs. ★~5k.","theCatch":"It tries to be four products at once, so expect a broad surface that overlaps with Langfuse, Opik and LiteLLM. The production self-host is more than a single Node process. The managed LangWatch Cloud is the commercial path, and README links lean that way.","verdict":"Worth a look if you want agent tests and an LLM gateway in the same tool as your traces.","addedAt":"2026-10-18","featured":false,"domain":"AI Agents","subSpecialty":"Agents","subcategory":"Evals & Tracing","tags":["self-host","permissive","coding-agents","evals","observability","testing","active"],"capabilities":[],"surfaces":["Self-host"],"ecosystem":["Multi-platform"],"licenseModel":"Apache-2.0","githubStars":4977,"starsPerWeek":31,"access":"free","archived":false,"repo":{"createdAt":"2023-09-09","pushedAt":"2026-10-19","license":"Apache-2.0","archived":false,"checkedAt":"2026-10-19"},"links":{"self":"/api/gems/langwatch","html":"/gems/langwatch","markdown":"/gems/langwatch.md"}},{"slug":"holmesgpt","name":"HolmesGPT","tagline":"Apache-2.0 CNCF AI agent for production incidents","category":"agents","url":"https://holmesgpt.dev","github":"https://github.com/HolmesGPT/holmesgpt","pricing":"open-source","hook":"HolmesGPT is an open-source AI agent that investigates production incidents: it pulls data from Kubernetes, VMs, cloud providers, databases and observability tools and proposes a root cause.","whatsGood":"Apache-2.0 (HolmesGPT/holmesgpt), release 0.45.0 (2026-10-15), a CNCF sandbox project originally from Robusta with major contributions from Microsoft. Run it as a CLI on an alert, or in the new Operator mode in Kubernetes for deployment verification and scheduled health checks that post findings to Slack and can open GitHub PRs. Works with your own LLM provider. ★~3.6k.","theCatch":"It needs read access to your infrastructure and observability data and sends that context to the LLM you configure, so pick the provider and scope permissions with care. Root-cause suggestions can be wrong and need human review, and letting it open PRs is a step to take deliberately. LLM costs scale with investigation volume.","verdict":"The most credible open SRE agent — start read-only and keep a human in the loop.","addedAt":"2026-10-18","featured":false,"domain":"AI Agents","subSpecialty":"Agents","subcategory":"Evals & Tracing","tags":["self-host","permissive","kubernetes","observability","active"],"capabilities":[],"surfaces":["Self-host"],"ecosystem":["Multi-platform"],"licenseModel":"Apache-2.0","githubStars":3626,"starsPerWeek":29,"access":"free","archived":false,"repo":{"createdAt":"2024-05-30","pushedAt":"2026-10-19","license":"Apache-2.0","archived":false,"checkedAt":"2026-10-19"},"links":{"self":"/api/gems/holmesgpt","html":"/gems/holmesgpt","markdown":"/gems/holmesgpt.md"}},{"slug":"laminar","name":"Laminar","tagline":"Apache-2.0 tracing, evals and signals for AI agents","category":"agents","url":"https://laminar.sh","github":"https://github.com/lmnr-ai/lmnr","pricing":"open-source","hook":"Laminar is an open-source observability platform for AI agents: OpenTelemetry-native tracing with one line of code, plain-English signals that alert on agent behaviour, evals, dashboards and SQL over your traces.","whatsGood":"Apache-2.0 (lmnr-ai/lmnr), release v0.3.2 (2026-10-16). Auto-traces the Vercel AI SDK, Browser Use, Stagehand, LangChain, OpenAI, Anthropic, Gemini and more. Signals let you describe behaviour like an agent stuck in a loop and get a Slack ping when it happens. An eval SDK and CLI run locally or in CI, and an MCP server and CLI let your coding agent query traces with SQL. ★~3.4k.","theCatch":"Self-hosting is a multi-service stack. Signals run an LLM over your agent runs, which adds cost and sends trace content to a model. The hosted Laminar cloud is the main product path. Crowded space next to Langfuse, Opik and LangWatch.","verdict":"Strongest pick when you trace browser and computer-use agents.","addedAt":"2026-10-18","featured":false,"domain":"AI Agents","subSpecialty":"Agents","subcategory":"Evals & Tracing","tags":["self-host","local-first","permissive","evals","observability","active"],"capabilities":[],"surfaces":["Self-host"],"ecosystem":["Multi-platform"],"licenseModel":"Apache-2.0","githubStars":3403,"starsPerWeek":30,"access":"free","archived":false,"repo":{"createdAt":"2024-08-29","pushedAt":"2026-10-18","license":"Apache-2.0","archived":false,"checkedAt":"2026-10-19"},"links":{"self":"/api/gems/laminar","html":"/gems/laminar","markdown":"/gems/laminar.md"}},{"slug":"openlit","name":"OpenLIT","tagline":"Apache-2.0 OTel tracing, evals and guardrails for agents","category":"agents","url":"https://openlit.io","github":"https://github.com/openlit/openlit","pricing":"open-source","hook":"OpenLIT is an open-source platform for observing and improving AI agents and coding agents with OpenTelemetry: tracing, evaluations, guardrails, prompt management, and cost and GPU monitoring.","whatsGood":"Apache-2.0 (openlit/openlit), free to self-host. Integrations cover Claude Code, Codex, Cursor, the OpenAI Agents SDK, LangGraph, CrewAI and 70+ more, with OpenTelemetry-native output you can send to your own backend. Includes an OTel GPU collector for model-serving hardware. ★~2.9k.","theCatch":"The README is heavy on the new agent-harness-engineering framing; underneath it is an OTel tracing and eval platform in a crowded field. Self-hosting means running its UI and ClickHouse. Many repos and releases (SDKs, collectors) to keep in sync.","verdict":"A solid self-hosted LLM observability option, especially if you also watch GPU cost.","addedAt":"2026-10-18","featured":false,"domain":"AI Agents","subSpecialty":"Agents","subcategory":"Evals & Tracing","tags":["self-host","permissive","coding-agents","evals","security","observability","active"],"capabilities":[],"surfaces":["Self-host"],"ecosystem":["Multi-platform"],"licenseModel":"Apache-2.0","githubStars":2858,"starsPerWeek":20,"access":"free","archived":false,"repo":{"createdAt":"2024-01-23","pushedAt":"2026-10-15","license":"Apache-2.0","archived":false,"checkedAt":"2026-10-19"},"links":{"self":"/api/gems/openlit","html":"/gems/openlit","markdown":"/gems/openlit.md"}},{"slug":"langsmith","name":"LangSmith","tagline":"LangChain freemium obs/eval/deploy — free Developer 5k traces/mo","category":"agents","url":"https://www.langchain.com/langsmith","github":"https://github.com/langchain-ai/langsmith-python","pricing":"freemium","hook":"LangSmith is LangChain’s agent observability, evaluation, and deployment platform — start on the free Developer seat, then pay as traces grow.","whatsGood":"Freemium (docs.smith.langchain.com / langchain.com/langsmith). Tracing, evals, prompt hubs, and deployment hooks wired for LangChain/LangGraph (and broader OpenTelemetry-style ingest). Developer plan: $0/seat with ~5k base traces/mo then pay-as-you-go; Plus/Enterprise for teams. SDKs are MIT: Python at langchain-ai/langsmith-python and JS/TS at langchain-ai/langsmith-javascript (both moved out of the langsmith-sdk repo, archived on 16 Oct 2026).","theCatch":"The product is hosted freemium (self-host exists on higher plans) — not a MIT self-host default like Langfuse. Vs Langfuse / Braintrust / Arize Phoenix: LangSmith is the LangChain-native commercial control plane; Langfuse is OSS self-host; Braintrust is eval-first; Phoenix is ELv2 Arize. Free tier is real but thin for busy agent fleets — budget traces early.","verdict":"Freemium LangChain obs/eval when the LangGraph stack wants one vendor pane — not Langfuse MIT self-host; free until ~5k traces.","addedAt":"2026-09-29","featured":false,"domain":"AI Agents","subSpecialty":"Agents","subcategory":"Evals & Tracing","tags":["free-tier","paid-led","permissive","evals","observability","active"],"capabilities":[],"surfaces":["Hosted"],"ecosystem":["Multi-platform"],"licenseModel":"Freemium","githubStars":13,"starsPerWeek":0,"access":"paid-led","archived":false,"repo":{"createdAt":"2026-02-20","pushedAt":"2026-10-17","license":"MIT","archived":false,"checkedAt":"2026-10-19"},"links":{"self":"/api/gems/langsmith","html":"/gems/langsmith","markdown":"/gems/langsmith.md"}}],"meta":{"total":9,"count":9,"offset":0,"limit":null,"sort":"curated","sorts":["curated","catalog","new","stars","trending","name"],"filters":{"subcategory":"evals-and-tracing"},"next":null,"categories":["agents","nextjs","python","shopify","databases","devops","design","observability","rust","payments","comms","notifications","libraries"],"tags":["mcp","cli","tui","api","sdk","desktop-app","web-ui","editor-plugin","self-host","local-first","free-tier","paid-only","paid-led","permissive","copyleft","source-available","license-unclear","coding-agents","local-llm","rag","memory","evals","browser-automation","scraping","sandbox","skills","voice","security","auth","payments","realtime","postgres","vector","kubernetes","containers","observability","testing","workflows","graphics-3d","git","active","stale","archived","young-repo","fast-stars"]}}