Scraping & crawling tools

Free / OSS · OSS/MIT · 23.5k GitHub stars

VerdictMIT anti-scraper PoW layer. Free forever; tune policies.

CatchPoW slows mass scrapers; determined headless solvers still pass. Can impede Archive/RSS unless you allowlist. Not Cloudflare WAF. Distinct from Traefik/Caddy proxies.

Free / OSS · Apache-2.0 · 7.8k GitHub stars

VerdictApache-2.0 agent browser API. Self-host Steel; pay Browserbase-shaped cloud only if you must.

CatchVs Browserbase (managed fleet), Stagehand (AI Playwright layer), Browser Use (LLM browser framework): Steel is the browser sandbox/API — bring your own agent logic. Self-host means Chrome resource cost; cloud is freemium/paid convenience. Anti-bot arms race still applies.

Free / OSS · BSD-3-Clause · 88.5k GitHub stars

VerdictBSD-3 adaptive scrape/crawl + MCP. Self-run framework vs Firecrawl-shaped APIs.

CatchVs Firecrawl-style hosted scrape APIs: Scrapling is a library/framework you run (plus MCP), not a metered scrape SaaS with a managed anti-bot fleet. You own proxies, blockers, and ops; MCP does not erase site ToS or legal risk. Not a general browser-agent product like Browser Use/Steel.

Free / OSS · Apache-2.0 · 85.8k GitHub stars

VerdictApache-2.0 LLM crawler. Markdown-for-agents vs Scrapling/Agent Reach/Browser Use/Stagehand.

CatchVs Scrapling (adaptive scrape/crawl + MCP), Agent Reach (public-web CLI eyes), Browser Use/Stagehand (browser agents): Crawl4AI is an LLM-oriented crawler/extraction library, not a hosted scrape API, not a site-specific eyes CLI, and not a Playwright agent. You own proxies, blockers, rate limits, and site ToS.

Free / OSS · Freemium · 193k GitHub stars

VerdictAGPL+cloud scrape API. Freemium credits vs Scrapling/Crawl4AI self-run; license and Free-tier limits are loud.

CatchAGPL-3.0 on the open engine — network copyleft if you modify and offer it as a service. cloud: Free is only 1k credits/mo with low concurrency; Hobby starts ~$16–19/mo for 5k credits — past that you pay. Vs Scrapling/Crawl4AI (self-run libraries) and Agent Reach (zero-API-fee eyes CLI): Firecrawl is API-first with a metered cloud; self-host trades SaaS bill for AGPL+ops. Distinct from Scrapling/Crawl4AI — do not treat as the same card.

Free / OSS · OSS/MIT · 6.9k GitHub stars

VerdictOpen MCP for agents on your TradingView Desktop. Paid TV sub required; chart interface layer, not a trading bot.

CatchYou need a valid paid TradingView subscription plus Desktop with remote debugging enabled — the bridge does not bypass paywalls. Undocumented Electron internals can break on any TradingView update. Vs playwright-mcp / BrowserSkill / Hyperbrowser MCP: this is desktop TradingView CDP, not a general browser or scrape API. Not a trade-execution bot.

Free / OSS · Freemium · 11.9k GitHub stars

VerdictMIT Apify Store MCP when agents need Actor-scale scraping — not Playwright browser MCP alone; budget Apify compute, not just the license.

CatchActor runs and Apify platform usage are billed — the MCP server is MIT, the data path is freemium/paid compute. Vs playwright-mcp / browser-use / firecrawl-class scrapers: those drive browsers or crawl APIs you own — Apify MCP’s catch is Store Actors as MCP tools. Legacy `/sse` endpoint removed — point clients at https://mcp.apify.com; pin which Actors agents may call.

Free / OSS · OSS/MIT · 249 GitHub stars

VerdictMIT GTM catalog MCP when agents need operator maps for growth tools — not Apify scrape Actors and not a static awesome-list dump.

CatchRead-only catalog MCP — it does not execute vendor APIs for you; you still need each tool’s own keys when a workflow says so. Vs Apify MCP / generic awesome-lists: Apify runs Store Actors; this catch is GTM tool+workflow markdown as MCP prompts/tools. Vendor accuracy drifts — treat entries as starting maps, pin commits if you fork.

Free / OSS · Apache-2.0 · 26.2k GitHub stars

VerdictThe default Node scraping framework when a one-off Playwright script has turned into a crawler — budget for proxies.

CatchNeeds Node 22.13 or newer. The Python version is a separate project with its own feature set, so do not assume parity. Heavy production crawling usually needs paid proxies to avoid blocks — the library rotates them, it does not supply them. Vs apify-mcp (gives an agent access to ready-made Apify scrapers) and playwright (the browser automation underneath): Crawlee is the crawler framework you build your own scraper on.

Free / OSS · AGPL-3.0 · 7.6k GitHub stars

VerdictThe quickest way to put Firecrawl in an agent — try keyless, add a key once you need crawling.

CatchWithout an API key only scrape, search and parse work, and they are rate-limited; crawl, map and agent need a key and paid credits. Vs firecrawl (the scraping engine and API itself, AGPL-3.0): this is just the MIT MCP bridge to it.

Free / OSS · Apache-2.0 · 13.9k GitHub stars

VerdictThe first thing to install on any Linux box you want to watch with Prometheus.

CatchIt only exposes metrics: you still need a Prometheus server (or a compatible scraper) to store them and something like Grafana to view and alert on them. Running it in a container needs host networking, the host PID namespace and a read-only bind mount of / with --path.rootfs, or it ends up measuring the container instead of the machine. Windows needs the separate windows_exporter, and NVIDIA GPUs need dcgm-exporter.

Free / OSS · OSS/MIT · 13.1k GitHub stars

VerdictThe most batteries-included Python browser framework — powerful, but use the stealth parts responsibly.

CatchA large share of the README is about bypassing bot detection and CAPTCHAs, which can break site terms — use it on sites you own or are allowed to automate. The API surface is huge, so pick one style and stick to it. For new JavaScript projects, Playwright is the more common default.

Paid-led · Freemium

VerdictFreemium public-record research API. MCP+SQL over an indexed corpus; not Firecrawl, Crawl4AI, or an OSS database.

CatchResearcher is non-commercial; Team starts from $2k. This is a hosted data product, not an OSS dump you can fork and self-host. Vs Firecrawl / Crawl4AI / Context7: Scry is indexed corpus + SQL, not page fetch, crawl, or documentation packing. Check corpus coverage, freshness, and commercial terms before treating a query as evidence.