web-scraping repositories
Trending repositories tagged web-scraping, ranked by trending score. Updated daily.
-
1
firecrawl/firecrawl
Knowledge infrastructure to make your agents smarter. Search, scrape and access 550+ data providers through one API.
GitHub · TypeScript · 205,406 stars · 10,742 forks · +2,082 stars in 24 hours · Trending score 5.92
-
2
D4Vinci/Scrapling
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
GitHub · Python · 104,687 stars · 10,932 forks · +448 stars in 24 hours · Trending score 5.23
-
3
browser-use/browser-use
Open-source Python library for AI browser agents, plus Browser Use Cloud: stealth browsers and a hosted agent API.
GitHub · Python · 121,631 stars · 13,587 forks · +179 stars in 24 hours · Trending score 4.40
-
4
ProxyShard/ShardBrowser
Free, open-source anti-detect browser launcher for web scraping and multi-accounting. By the ShardX team (formerly ProxyShard). Engine-level fingerprint spoofing in Chromium 158 (WebGL / WebGPU / Client Hints / fonts / TLS), 220+ device profiles bundled, stable QUIC + WebRTC over SOCKS5.
GitHub · TypeScript · 1,704 stars · 240 forks · +270 stars in 24 hours · Trending score 4.36
-
5
lexmount/moli
Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust
GitHub · Rust · 24,165 stars · 693 forks · +140 stars in 24 hours · Trending score 4.11
-
6
cporter202/google-maps-scraping-apis
A powerhouse collection of Google Maps scraping APIs you can start using immediately to build local lead lists, monitor reviews, and power location intelligence. One of the most valuable Maps data lists on GitHub.
GitHub · 179 stars · 38 forks · +141 stars in 24 hours · Trending score 3.99
-
7
JCodesMore/ai-website-cloner-template
Clone any website with one command using AI coding agents
GitHub · TypeScript · 39,567 stars · 5,895 forks · +98 stars in 24 hours · Trending score 3.95
-
8
3441293738/creatorhub
多平台内容监控·采集·搬运 一个 Web 面板管起抖音 / 小红书 / 快手 / 视频号 / TikTok / 哔哩哔哩 / Instagram / 海外小红书(rednote)
GitHub · Python · 3,726 stars · 731 forks · +54 stars in 24 hours · Trending score 3.82
-
9
unclecode/crawl4ai
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
GitHub · Python · 87,614 stars · 9,134 forks · +70 stars in 24 hours · Trending score 3.71
-
10
jo-inc/camofox-browser
Stealth headless browser for AI agents — bypass Cloudflare, bot detection, and anti-scraping. Drop-in Puppeteer/Playwright replacement.
GitHub · JavaScript · 12,432 stars · 1,271 forks · +58 stars in 24 hours · Trending score 3.39
-
11
scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
GitHub · Python · 65,517 stars · 12,150 forks · +40 stars in 24 hours · Trending score 3.26
-
12
tiliondev/fortress
Stealth Chromium engine that stops scrapers and browser agents from getting blocked, with one line of code change.
GitHub · Python · 810 stars · 55 forks · +86 stars in 24 hours · Trending score 3.22
-
13
dgtlmoon/changedetection.io
Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock alerts, and website defacement monitoring— MCP included! all for free or enjoy our SaaS plan!
GitHub · Python · 35,930 stars · 2,265 forks · +32 stars in 24 hours · Trending score 3.13
-
14
CloakHQ/CloakBrowser
Stealth Chromium that passes every bot detection test. Drop-in Playwright replacement with source-level fingerprint patches. 50+ tests passed.
GitHub · Python · 33,003 stars · 2,740 forks · +28 stars in 24 hours · Trending score 3.01
-
15
getmaxun/maxun
Turn any website into a structured API. Extract, automate, search and monitor the web.
GitHub · TypeScript · 18,533 stars · 1,597 forks · +17 stars in 24 hours · Trending score 3.00
-
16
apify/crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
GitHub · TypeScript · 26,505 stars · 1,774 forks · +19 stars in 24 hours · Trending score 2.91
-
17
guohuiyuan/go-novel-dl
go-novel-dl 是 Go 语言开发的多源小说下载器,支持 CLI 与 Web 界面,可跨站点并发搜索、一键下载并导出,已适配 ESJZone、SFACG、刺猬猫、若初文学等多个书源。novel-dl交流群:1094455212
GitHub · Go · 337 stars · 32 forks · +34 stars in 24 hours · Trending score 2.80
-
18
SeraphinaGlacia/steam-scraper
一个简单、高效的 Steam 评论数据爬虫。爬取 Steam 游戏详情和评论历史趋势到 SQLite / CSV。
GitHub · Python · 43 stars · 2 forks · +34 stars in 24 hours · Trending score 2.74
-
19
browserbase/stagehand
The SDK to extract data and interact with any site on the web. Get started with Claude Code, Codex, Eve, Mastra, and more.
GitHub · TypeScript · 26,106 stars · 1,836 forks · +14 stars in 24 hours · Trending score 2.65
-
20
ovobrowsersupport/ovobrowser
源码级 Chromium 指纹内核,检测站按普通 Chrome 评分。Source-level Chromium fingerprint kernel that passes bot detection tests. Not JS injection — C++ patches compiled into the binary.
GitHub · 1,274 stars · 71 forks · +19 stars in 24 hours · Trending score 2.58
-
21
spider-rs/spider
Foundational low latency web data collecting in Rust
GitHub · Rust · 2,855 stars · 331 forks · +23 stars in 24 hours · Trending score 2.47
-
22
oss-invisible/playwright
invisible playwright: Playwright that anti-bots cannot see, so no captchas: same API, anti-detect stealth headless Firefox, undetected fingerprint, bypass bot detection, Python web scraping and browser automation.
GitHub · Python · 2,612 stars · 279 forks · +20 stars in 24 hours · Trending score 2.36
-
23
fastcrw/crw
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.
GitHub · Rust · 1,283 stars · 102 forks · +16 stars in 24 hours · Trending score 2.35
-
24
vibheksoni/stealth-browser-mcp
Stealth browser automation for AI agents over MCP and the Chrome DevTools Protocol: navigation, network hooks, DOM extraction, and pixel-accurate UI cloning.
GitHub · Python · 3,382 stars · 416 forks · +14 stars in 24 hours · Trending score 2.27