Overlap
Memory & Evals for Python
Memory & Evals tools that run on Python. The Catch for each, not a winner list.
6 tools · the Catch, not a ranking
- DeepEvalApache-2.0 pytest-style LLM/agent evals; Confident AI is the paid cloud
OSS is the runner — collaboration/dashboards/monitoring are Confident AI (Free: 2 seats/1 project/5 runs/wk; Starter $200/mo; Team $2k/mo). LLM-as-judge burns provider tokens. Vs Ragas: broader agent/chat coverage, not RAG-only.
- GraphitiApache-2.0 temporal knowledge graphs for agent memory; Zep's OSS engine
Framework only — you run a graph DB + ops. Zep Community Edition deprecated; full Zep SaaS is separate (Free credits then Flex ~$104/mo billed annually). Not a drop-in chat history store. Schema/ops cost vs Mem0 Docker.
- OpenVikingAGPL self-evolving context database for memory, RAG, and skills
AGPL-3.0 is strong copyleft, including the network-copyleft boundary when you modify OpenViking and offer it as a service. Get legal review before exposing a modified deployment over a network. Vs Mem0, Graphiti, and Letta: OpenViking is the context database spanning memory/RAG/skills, not simply a memory API, graph memory layer, or agent runtime.
- HindsightMIT retain→recall→reflect agent memory (+ MCP)
Vs Mem0, Graphiti, Engram, and Supermemory: Hindsight is a retain/recall/reflect memory product with optional hosted Cloud/Enterprise, not a general memory SDK alone, a temporal knowledge graph, a local binary brain, or a speed-first memory API. Treat LongMemEval / SOTA marketing as vendor claims, not catalog proof; Cloud and Enterprise are upsell paths beside the OSS self-host bar.
- CogneeApache-2.0 AI memory platform — local knowledge graph for agents
Vs Mem0 / Hindsight: Mem0 is freemium memory infra SDK/cloud; Hindsight is MIT retain→recall→reflect (+ MCP). Cognee’s catch is KG-first company/agent memory you self-host — not Mem0’s hosted memory API and not Hindsight’s reflect loop alone. Graph + embedding pipelines cost CPU/disk; hosted/cloud extras (if any) are optional beside the OSS bar.
- iFixAiApache-2.0 agents eval/audit — pip install, premium ≠ paywall
Vs Braintrust / Langfuse / Ragas / Promptfoo: Braintrust is eval-first commercial; Langfuse is MIT obs/self-host; Ragas is Apache RAG/agent metrics + CI scorers; Promptfoo is OSS prompt/agent eval — iFixAi’s catch is agents eval/audit as an installable audit kit, not Braintrust seats, not Langfuse traces, not Ragas metrics alone, not Promptfoo prompt suites. Stars grow fast (~47 a week): judge the audit loop, not the counter.