crawl repositories

Trending repositories tagged crawl, ranked by trending score. Updated daily.

  1. 1
    tavily-ai/tavily-python

    The Tavily Python SDK allows for easy interaction with the Tavily API, offering the full range of our search, extract, crawl, map, and research functionalities directly from your Python programs. Easily integrate smart search, content extraction, and research capabilities into your applications, harnessing Tavily's powerful features.

    GitHub · Python · 1,432 stars · 190 forks · +14 stars in 24 hours · Trending score 1.76

  2. 2
    tavily-ai/tavily-n8n-node

    A node for n8n that integrates the Tavily API, enabling powerful web search and content extraction within your no-code automation workflows.

    GitHub · TypeScript · 23 stars · 8 forks · +2 stars in 24 hours · Trending score 0.92

  3. 3
    ArchiveTeam/wget-lua

    Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

    GitHub · C · 140 stars · 19 forks · +1 stars in 24 hours · Trending score 0.45

  4. 4
    scrapfly/Antibot-Detector

    Real-time detection of anti-bot systems, captchas & fingerprinting techniques. Identifies Cloudflare, Akamai, DataDome, reCAPTCHA, hCaptcha, Shape Security & more with confidence scoring and advanced capture tools.

    GitHub · JavaScript · 632 stars · 74 forks · 0 stars in 24 hours · Trending score 0.44

  5. 5
    coder-hxl/x-crawl

    Flexible Node.js crawler library

    GitHub · TypeScript · 1,888 stars · 115 forks · −2 stars in 24 hours · Trending score 0.10

  6. 6
    any4ai/AnyCrawl

    AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.

    GitHub · TypeScript · 3,528 stars · 372 forks · 0 stars in 24 hours · Trending score 0.03

  7. 7
    tavily-ai/tavily-js

    Official TypeScript/JavaScript library for Tavily.

    GitHub · TypeScript · 95 stars · 10 forks · 0 stars in 24 hours · Trending score 0.00

  8. 8
    mangenotwork/gathertool

    gathertool是golang脚本化开发库,目的是提高对应场景程序开发的效率;轻量级爬虫库,接口测试&压力测试库,DB操作库等。

    GitHub · Go · 55 stars · 15 forks · 0 stars in 24 hours · Trending score 0.00

  9. 9
    spatie/laravel-site-search

    Create a full-text search index by crawling your site

    GitHub · PHP · 310 stars · 25 forks · 0 stars in 24 hours · Trending score 0.00