crawl repositories
Trending repositories tagged crawl, ranked by trending score. Updated daily.
-
1
tavily-ai/tavily-python
The Tavily Python SDK allows for easy interaction with the Tavily API, offering the full range of our search, extract, crawl, map, and research functionalities directly from your Python programs. Easily integrate smart search, content extraction, and research capabilities into your applications, harnessing Tavily's powerful features.
GitHub · Python · 1,432 stars · 190 forks · +14 stars in 24 hours · Trending score 1.76
-
2
tavily-ai/tavily-n8n-node
A node for n8n that integrates the Tavily API, enabling powerful web search and content extraction within your no-code automation workflows.
GitHub · TypeScript · 23 stars · 8 forks · +2 stars in 24 hours · Trending score 0.92
-
3
ArchiveTeam/wget-lua
Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.
GitHub · C · 140 stars · 19 forks · +1 stars in 24 hours · Trending score 0.45
-
4
scrapfly/Antibot-Detector
Real-time detection of anti-bot systems, captchas & fingerprinting techniques. Identifies Cloudflare, Akamai, DataDome, reCAPTCHA, hCaptcha, Shape Security & more with confidence scoring and advanced capture tools.
GitHub · JavaScript · 632 stars · 74 forks · 0 stars in 24 hours · Trending score 0.44
-
5
coder-hxl/x-crawl
Flexible Node.js crawler library
GitHub · TypeScript · 1,888 stars · 115 forks · −2 stars in 24 hours · Trending score 0.10
-
6
any4ai/AnyCrawl
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
GitHub · TypeScript · 3,528 stars · 372 forks · 0 stars in 24 hours · Trending score 0.03
-
7
tavily-ai/tavily-js
Official TypeScript/JavaScript library for Tavily.
GitHub · TypeScript · 95 stars · 10 forks · 0 stars in 24 hours · Trending score 0.00
-
8
mangenotwork/gathertool
gathertool是golang脚本化开发库,目的是提高对应场景程序开发的效率;轻量级爬虫库,接口测试&压力测试库,DB操作库等。
GitHub · Go · 55 stars · 15 forks · 0 stars in 24 hours · Trending score 0.00
-
9
spatie/laravel-site-search
Create a full-text search index by crawling your site
GitHub · PHP · 310 stars · 25 forks · 0 stars in 24 hours · Trending score 0.00