Crawl4AI is an open-source Python web crawler that converts pages into Markdown and structured data, using a real browser so JavaScript-heavy sites work. It offers Markdown output with content filters, CSS and XPath extraction that needs no LLM, LLM-based extraction into JSON schemas, and deep crawling in breadth-first, depth-first or best-first order.
Install the crawl4ai package with pip, run its setup command to install the browser, then use the async crawler and call its run method with a URL.
Apache-2.0 licensed and free to self-host, which makes it a common alternative to paid crawling APIs.