{"data":{"slug":"crawl4ai","name":"Crawl4AI","tagline":"Apache-2.0 LLM-friendly crawler — markdown for agents","category":"agents","categoryName":"AI Agents","url":"https://crawl4ai.com","github":"https://github.com/unclecode/crawl4ai","pricing":"open-source","hook":"Crawl4AI turns the web into clean, LLM-ready markdown — a crawler shaped for RAG and agents, not a generic scrape SDK.","whatsGood":"Apache-2.0 (unclecode/crawl4ai). Async Python crawler tuned for LLM pipelines: markdown extraction, chunking hooks, and agent-friendly output. Site at crawl4ai.com; pip-installable. Huge adoption (~84k★).","theCatch":"Vs Scrapling (adaptive scrape/crawl + MCP), Agent Reach (public-web CLI eyes), Browser Use/Stagehand (browser agents): Crawl4AI is an LLM-oriented crawler/extraction library, not a hosted scrape API, not a site-specific eyes CLI, and not a Playwright agent. You own proxies, blockers, rate limits, and site ToS.","verdict":"Apache-2.0 LLM crawler. Markdown-for-agents vs Scrapling/Agent Reach/Browser Use/Stagehand.","addedAt":"2026-09-16","featured":false,"domain":"AI Agents","subSpecialty":"Agents","subcategory":"Scraping for LLMs","tags":["sdk","self-host","permissive","rag","scraping","active","fast-stars"],"capabilities":["sdk"],"surfaces":["SDK","Self-host"],"ecosystem":["Multi-platform"],"licenseModel":"Apache-2.0","githubStars":85801,"starsPerWeek":672,"access":"free","archived":false,"repo":{"createdAt":"2024-05-09","pushedAt":"2026-10-19","license":"Apache-2.0","archived":false,"checkedAt":"2026-10-19"}},"related":[{"slug":"claude-code","name":"Claude Code","tagline":"Terminal-first coding agent with deep repository context"},{"slug":"cursor","name":"Cursor","tagline":"The AI-first code editor that understands your codebase"},{"slug":"devin","name":"Devin","tagline":"Autonomous AI engineer for defined engineering tasks"},{"slug":"stagehand","name":"Stagehand","tagline":"AI-native browser automation that mixes code and natural language"},{"slug":"browser-use","name":"Browser Use","tagline":"Open-source framework for LLM-driven browser agents"}],"links":{"self":"/api/gems/crawl4ai","html":"/gems/crawl4ai","markdown":"/gems/crawl4ai.md","category":"/api/gems?category=agents","categoryPage":"/stacks/agents"}}