Web Search & Data Extraction
Search, crawling and document-parsing APIs that give LLMs and agents fresh web data and clean, structured text from pages and files, so answers come from live sources instead of stale training data.
-
Tavily
Search API that returns short, ranked, ready-to-use results for LLM agents, with extract and crawl endpoints too. A common default search tool in agent frameworks.
-
Brave Search API
Web search API running on Brave's own index instead of reselling Google or Bing. Includes endpoints that return page snippets sized for LLM context.
-
Jina AI
Embeddings, rerankers and the Reader API that turns URLs into Markdown for LLMs. Acquired by Elastic in October 2025, with models also offered through Elastic Cloud.
-
Firecrawl
Turns any website into clean markdown or JSON. Use it to feed live web pages to an LLM or build a RAG index. Self-host the open-source version or call the hosted API.
-
Exa
Search API built for agents. It finds pages by meaning rather than keywords and can return full page text, which suits research tools and RAG.
-
Crawl4AI
Free, self-hosted Python crawler that turns pages into clean markdown for LLMs. A good fit when you want Firecrawl-style output without paying for an API.
-
Apify
Marketplace of ready-made scrapers (called Actors) for sites like Google Maps, LinkedIn and Amazon, plus a platform to run your own. Has an MCP server for agents.
-
Unstructured
Converts PDFs, Word files, slides and HTML into clean, chunked elements for RAG. Open-source library, with a hosted API for heavier workloads.
-
Docling
Open-source document converter from IBM Research. Turns PDFs, DOCX and slides into markdown or JSON and keeps tables and layout intact. Runs locally.
-
LlamaParse
LlamaIndex's hosted parser for messy documents such as scanned PDFs, tables and charts. Priced per page and plugs straight into LlamaIndex pipelines.