Crawl4ai

Crawl4AI is the open-source Firecrawl alternative for AI-optimized web crawling — run locally (no API, no cost), crawl any URL to clean markdown, extract structured data with CSS/XPath/LLM strategies, handle JavaScript-rendered pages via Playwright, and deep-crawl entire sites by following links. For agents: crawl a documentation site, extract clean markdown for RAG ingestion, use LLMExtractionStrategy to pull structured schema from any page, and deep-crawl with link following for comprehensive content capture. Zero network dependencies — pure local execution. 25k+ GitHub stars. Confidence is docs-derived.

Python SDK: pip install "crawl4ai[all]". Basic crawl: import asyncio; from crawl4ai import AsyncWebCrawler; async def main(): async with AsyncWebCrawler() as crawler: result = await crawler.arun(url="https://example.com"); print(result.markdown); asyncio.run(main()). Result fields: result.markdown (clean text), result.html (raw HTML), result.links (extracted links), result.media (images, videos). CSS extraction: from crawl4ai.extraction_strategy import JsonCssExtractionStrategy; schema = {"name": "Articles", "baseSelector": ".article", "fields": [{"name": "title", "selector": "h2", "type": "text"}, {"name": "url", "selector": "a", "type": "attribute", "attribute": "href"}]}; strategy = JsonCssExtractionStrategy(schema). LLM extraction: LLMExtractionStrategy(provider="openai/gpt-4o", api_token="...", schema=MyPydanticModel.schema()). Deep crawl: result = await crawler.arun(url="...", deep_crawl=True, max_pages=50). JS execution: result = await crawler.arun(url="...", js_code="window.scrollTo(0, document.body.scrollHeight)")

No external auth — Crawl4AI runs fully locally using Playwright for browser automation. No API key or network calls to Crawl4AI servers. LLM extraction: if using LLMExtractionStrategy, LLM provider API key required (OpenAI, Anthropic, etc.) — passed per-request. Rate limiting: your own responsibility (add delays between requests for politeness). robots.txt: Crawl4AI does not enforce robots.txt by default — implement compliance in your crawler logic. Self-hosted only — no SaaS option.

JS rendering: Playwright-based — handles SPAs, lazy loading, infinite scroll. Page wait: configure wait_for="css:.content-loaded" to wait for elements. Screenshot: result.screenshot for visual debugging. Concurrent crawling: AsyncWebCrawler supports multiple simultaneous crawls. Memory: each browser context uses ~100MB RAM — limit concurrency for memory-constrained environments. Anti-bot: some sites detect headless browsers — use stealth mode (playwright_extra plugins). Large sites: deep crawl with max_pages cap; use sitemap parsing for structured discovery.

docs.crawl4ai.com covers async crawler API, extraction strategies (CSS, XPath, LLM), deep crawl configuration, and Docker deployment. Getting started: pip install crawl4ai[all], crawl4ai-setup — first crawl in 3 minutes. Fully free and open-source. GitHub: unclecode/crawl4ai (25k+ stars). Community via Discord (very active). Excellent documentation with AI-focused examples. One of the fastest-growing web scraping libraries for AI use cases.