web-scraping repositories

Trending repositories tagged web-scraping, ranked by trending score. Updated daily.

  1. 1
    firecrawl/firecrawl

    Knowledge infrastructure to make your agents smarter. Search, scrape and access 550+ data providers through one API.

    GitHub · TypeScript · 205,406 stars · 10,742 forks · +2,082 stars in 24 hours · Trending score 5.92

  2. 2
    D4Vinci/Scrapling

    🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

    GitHub · Python · 104,687 stars · 10,932 forks · +448 stars in 24 hours · Trending score 5.23

  3. 3
    browser-use/browser-use

    Open-source Python library for AI browser agents, plus Browser Use Cloud: stealth browsers and a hosted agent API.

    GitHub · Python · 121,631 stars · 13,587 forks · +179 stars in 24 hours · Trending score 4.40

  4. 4
    ProxyShard/ShardBrowser

    Free, open-source anti-detect browser launcher for web scraping and multi-accounting. By the ShardX team (formerly ProxyShard). Engine-level fingerprint spoofing in Chromium 158 (WebGL / WebGPU / Client Hints / fonts / TLS), 220+ device profiles bundled, stable QUIC + WebRTC over SOCKS5.

    GitHub · TypeScript · 1,704 stars · 240 forks · +270 stars in 24 hours · Trending score 4.36

  5. 5
    lexmount/moli

    Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust

    GitHub · Rust · 24,165 stars · 693 forks · +140 stars in 24 hours · Trending score 4.11

  6. 6
    cporter202/google-maps-scraping-apis

    A powerhouse collection of Google Maps scraping APIs you can start using immediately to build local lead lists, monitor reviews, and power location intelligence. One of the most valuable Maps data lists on GitHub.

    GitHub · 179 stars · 38 forks · +141 stars in 24 hours · Trending score 3.99

  7. 7
    JCodesMore/ai-website-cloner-template

    Clone any website with one command using AI coding agents

    GitHub · TypeScript · 39,567 stars · 5,895 forks · +98 stars in 24 hours · Trending score 3.95

  8. 8
    3441293738/creatorhub

    多平台内容监控·采集·搬运 一个 Web 面板管起抖音 / 小红书 / 快手 / 视频号 / TikTok / 哔哩哔哩 / Instagram / 海外小红书(rednote)

    GitHub · Python · 3,726 stars · 731 forks · +54 stars in 24 hours · Trending score 3.82

  9. 9
    unclecode/crawl4ai

    Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

    GitHub · Python · 87,614 stars · 9,134 forks · +70 stars in 24 hours · Trending score 3.71

  10. 10
    jo-inc/camofox-browser

    Stealth headless browser for AI agents — bypass Cloudflare, bot detection, and anti-scraping. Drop-in Puppeteer/Playwright replacement.

    GitHub · JavaScript · 12,432 stars · 1,271 forks · +58 stars in 24 hours · Trending score 3.39

  11. 11
    scrapy/scrapy

    Scrapy, a fast high-level web crawling & scraping framework for Python.

    GitHub · Python · 65,517 stars · 12,150 forks · +40 stars in 24 hours · Trending score 3.26

  12. 12
    tiliondev/fortress

    Stealth Chromium engine that stops scrapers and browser agents from getting blocked, with one line of code change.

    GitHub · Python · 810 stars · 55 forks · +86 stars in 24 hours · Trending score 3.22

  13. 13
    dgtlmoon/changedetection.io

    Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock alerts, and website defacement monitoring— MCP included! all for free or enjoy our SaaS plan!

    GitHub · Python · 35,930 stars · 2,265 forks · +32 stars in 24 hours · Trending score 3.13

  14. 14
    CloakHQ/CloakBrowser

    Stealth Chromium that passes every bot detection test. Drop-in Playwright replacement with source-level fingerprint patches. 50+ tests passed.

    GitHub · Python · 33,003 stars · 2,740 forks · +28 stars in 24 hours · Trending score 3.01

  15. 15
    getmaxun/maxun

    Turn any website into a structured API. Extract, automate, search and monitor the web.

    GitHub · TypeScript · 18,533 stars · 1,597 forks · +17 stars in 24 hours · Trending score 3.00

  16. 16
    apify/crawlee

    Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

    GitHub · TypeScript · 26,505 stars · 1,774 forks · +19 stars in 24 hours · Trending score 2.91

  17. 17
    guohuiyuan/go-novel-dl

    go-novel-dl 是 Go 语言开发的多源小说下载器,支持 CLI 与 Web 界面,可跨站点并发搜索、一键下载并导出,已适配 ESJZone、SFACG、刺猬猫、若初文学等多个书源。novel-dl交流群:1094455212

    GitHub · Go · 337 stars · 32 forks · +34 stars in 24 hours · Trending score 2.80

  18. 18
    SeraphinaGlacia/steam-scraper

    一个简单、高效的 Steam 评论数据爬虫。爬取 Steam 游戏详情和评论历史趋势到 SQLite / CSV。

    GitHub · Python · 43 stars · 2 forks · +34 stars in 24 hours · Trending score 2.74

  19. 19
    browserbase/stagehand

    The SDK to extract data and interact with any site on the web. Get started with Claude Code, Codex, Eve, Mastra, and more.

    GitHub · TypeScript · 26,106 stars · 1,836 forks · +14 stars in 24 hours · Trending score 2.65

  20. 20
    ovobrowsersupport/ovobrowser

    源码级 Chromium 指纹内核,检测站按普通 Chrome 评分。Source-level Chromium fingerprint kernel that passes bot detection tests. Not JS injection — C++ patches compiled into the binary.

    GitHub · 1,274 stars · 71 forks · +19 stars in 24 hours · Trending score 2.58

  21. 21
    spider-rs/spider

    Foundational low latency web data collecting in Rust

    GitHub · Rust · 2,855 stars · 331 forks · +23 stars in 24 hours · Trending score 2.47

  22. 22
    oss-invisible/playwright

    invisible playwright: Playwright that anti-bots cannot see, so no captchas: same API, anti-detect stealth headless Firefox, undetected fingerprint, bypass bot detection, Python web scraping and browser automation.

    GitHub · Python · 2,612 stars · 279 forks · +20 stars in 24 hours · Trending score 2.36

  23. 23
    fastcrw/crw

    Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.

    GitHub · Rust · 1,283 stars · 102 forks · +16 stars in 24 hours · Trending score 2.35

  24. 24
    vibheksoni/stealth-browser-mcp

    Stealth browser automation for AI agents over MCP and the Chrome DevTools Protocol: navigation, network hooks, DOM extraction, and pixel-accurate UI cloning.

    GitHub · Python · 3,382 stars · 416 forks · +14 stars in 24 hours · Trending score 2.27