ScrapeGraphAI

ScrapeGraphAI is a Python library for AI-powered web scraping: instead of writing brittle XPath selectors, you describe the data you want in natural language and it builds a scraping pipeline (a graph of nodes: fetch, parse, generate, merge) that returns structured JSON. It supports single pages, search-augmented scraping, and multi-page crawls, and it can run against local models via Ollama or any major provider API.

With ~31,800 GitHub stars it is one of the most popular scraping libraries of the LLM era, and the graph-based pipeline design makes complex extractions composable rather than one-shot prompt hacks. It also ships prebuilt integrations and a cloud offering for managed runs.

The honest caveat: LLM scraping costs tokens per page, so at scale it is far more expensive than traditional scrapers — use it where layout variability defeats selectors, not as a blanket replacement. And like all scraping tools, it doesn't exempt you from a site's terms of service or robots.txt.

1. pip install scrapegraphai (Python 3.8+).
2. Set your model provider key (OpenAI/Anthropic/etc.) or point it at a local Ollama model.
3. Define a prompt describing the fields you want and call SmartScraperGraph(prompt, url).
4. For multi-page or search-augmented jobs, use SearchGraph or compose custom node graphs.