ai-scraping repositories
Trending repositories tagged ai-scraping, ranked by trending score. Updated daily.
-
1
firecrawl/firecrawl
Knowledge infrastructure to make your agents smarter. Search, scrape and access 550+ data providers through one API.
GitHub · TypeScript · 206,734 stars · 10,802 forks · +2,005 stars in 24 hours · Trending score 5.95
-
2
D4Vinci/Scrapling
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
GitHub · Python · 104,965 stars · 10,981 forks · +411 stars in 24 hours · Trending score 5.15
-
3
itsOwen/CyberScraper-2077
A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama
GitHub · Python · 3,302 stars · 360 forks · +14 stars in 24 hours · Trending score 1.76
-
4
WeebDataHoarder/go-away
[Mirror] Self-hosted abuse detection and rule enforcement against low-effort mass AI scraping and bots.
GitHub · Go · 222 stars · 13 forks · +10 stars in 24 hours · Trending score 1.66
-
5
ArchiveBox/abx-dl
⬇️ A simple all-in-one CLI tool to download EVERYTHING from a URL (like youtube-dl/yt-dlp, forum-dl, gallery-dl, simpler ArchiveBox). 🎭 Uses headless Chrome to get HTML, JS, CSS, images/video/audio/subtitles, PDFs, screenshots, article text, git repos, and more...
GitHub · Python · 154 stars · 10 forks · +1 stars in 24 hours · Trending score 0.69
-
6
0xMassi/webclaw
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
GitHub · Rust · 2,442 stars · 245 forks · −3 stars in 24 hours · Trending score 0.10
-
7
any4ai/AnyCrawl
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
GitHub · TypeScript · 3,528 stars · 372 forks · 0 stars in 24 hours · Trending score 0.03
-
8
ScrapeGraphAI/Scrapegraph-ai
Python scraper based on AI
GitHub · Python · 32,889 stars · 3,360 forks · 0 stars in 24 hours · Trending score 0.00