web-crawling repositories

Trending repositories tagged web-crawling, ranked by trending score. Updated daily.

  1. 1
    lexmount/moli

    Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust

    GitHub · Rust · 24,146 stars · 694 forks · +161 stars in 24 hours · Trending score 4.19

  2. 2
    unclecode/crawl4ai

    Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

    GitHub · Python · 87,596 stars · 9,143 forks · +87 stars in 24 hours · Trending score 3.84

  3. 3
    apify/crawlee

    Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

    GitHub · TypeScript · 26,505 stars · 1,774 forks · +40 stars in 24 hours · Trending score 3.23

  4. 4
    tiliondev/fortress

    Stealth Chromium engine that stops scrapers and browser agents from getting blocked, with one line of code change.

    GitHub · Python · 810 stars · 55 forks · +86 stars in 24 hours · Trending score 3.22

  5. 5
    apify/crawlee-python

    Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.

    GitHub · Python · 9,680 stars · 837 forks · +3 stars in 24 hours · Trending score 1.59

  6. 6
    MontFerret/ferret

    Declarative data automation language and Go runtime for structured extraction workflows.

    GitHub · Go · 6,026 stars · 327 forks · +6 stars in 24 hours · Trending score 1.51

  7. 7
    geonodecom/cli

    Geonode CLI: Web Scraping, Crawling & Search. Extract pages, crawl websites, map URLs and search the web from your terminal, with JSON output for scripts and scheduled data pipelines.

    GitHub · Python · 24 stars · 0 forks · +9 stars in 24 hours · Trending score 1.11

  8. 8
    superagents-lab/search1api-mcp

    Official Search1API MCP server for web search, news, crawling, sitemaps, and trends—hosted with OAuth 2.1 or local via npm.

    GitHub · TypeScript · 175 stars · 38 forks · +2 stars in 24 hours · Trending score 0.81

  9. 9
    brightdata/brightdata-mcp

    A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.

    GitHub · JavaScript · 2,724 stars · 337 forks · 0 stars in 24 hours · Trending score 0.69

  10. 10
    bunoriapp/bunori

    Read, Manage, Download & Export Light Novels, all in one Android app.

    GitHub · Kotlin · 48 stars · 8 forks · 0 stars in 24 hours · Trending score 0.27

  11. 11
    apify/apify-sdk-python

    Apify SDK for Python—The official library for building Apify Actors: serverless cloud programs for web scraping, browser automation, data processing, and AI agents. Manages the Actor lifecycle, storages (datasets, key-value stores, request queues), events, proxies, and pay-per-event monetization. Built on top of the the Apify API Client.

    GitHub · Python · 180 stars · 32 forks · 0 stars in 24 hours · Trending score 0.25

  12. 12
    lexmount/webstone

    Benchmark browsers and web tools for AI agents: Browser Trajectory Replay, Web Fetch / Search / Extract / Scraping / Crawling.

    GitHub · Python · 10 stars · 1 forks · 0 stars in 24 hours · Trending score 0.09

  13. 13
    apify/apify-client-python

    Apify API client for Python—Programmatically run Actors, manage and stream data from storages (datasets, key-value stores, request queues), schedule and monitor runs, and access the full Apify platform API. Sync and async interfaces with automatic retries and pagination.

    GitHub · Python · 105 stars · 27 forks · 0 stars in 24 hours · Trending score 0.03

  14. 14
    kkrugler/flink-crawler

    Continuous scalable web crawler built on top of Flink and crawler-commons

    GitHub · Java · 53 stars · 18 forks · 0 stars in 24 hours · Trending score 0.00

  15. 15
    jgujerry/python-frameworks

    Another curated list of Python frameworks

    GitHub · HTML · 63 stars · 6 forks · 0 stars in 24 hours · Trending score 0.00

  16. 16
    romek-rozen/cc-crawl4ai

    Native non-MCP Crawl4AI and Trafilatura plugin for Claude Code

    GitHub · Python · 12 stars · 1 forks · 0 stars in 24 hours · Trending score 0.00

  17. 17
    Zulwatha/content-parity

    Measure what a website serves to machine readers. Fetches any page as several honest identities, including a cryptographically signed agent, and reports what each one received.

    GitHub · Go · 25 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00