web-crawler repositories
Trending repositories tagged web-crawler, ranked by trending score. Updated daily.
-
1
firecrawl/firecrawl
Knowledge infrastructure to make your agents smarter. Search, scrape and access 550+ data providers through one API.
GitHub · TypeScript · 205,874 stars · 10,762 forks · +2,082 stars in 24 hours · Trending score 5.92
-
2
lexmount/moli
Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust
GitHub · Rust · 24,196 stars · 693 forks · +140 stars in 24 hours · Trending score 4.11
-
3
unclecode/crawl4ai
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
GitHub · Python · 87,632 stars · 9,137 forks · +70 stars in 24 hours · Trending score 3.71
-
4
apify/crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
GitHub · TypeScript · 26,509 stars · 1,774 forks · +19 stars in 24 hours · Trending score 2.91
-
5
spider-rs/spider
Foundational low latency web data collecting in Rust
GitHub · Rust · 2,856 stars · 332 forks · +23 stars in 24 hours · Trending score 2.47
-
6
fastcrw/crw
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.
GitHub · Rust · 1,283 stars · 102 forks · +16 stars in 24 hours · Trending score 2.35
-
7
firecrawl/firecrawl-mcp-server
Official Firecrawl MCP server, the context API for AI agents: search, scrape, and interact with the web, plus parse, crawl, map, Agent and Monitor, from any MCP client.
GitHub · JavaScript · 7,872 stars · 942 forks · +5 stars in 24 hours · Trending score 2.02
-
8
KnockOutEZ/wigolo
The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.
GitHub · TypeScript · 5,558 stars · 464 forks · +8 stars in 24 hours · Trending score 1.89
-
9
apify/crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.
GitHub · Python · 9,680 stars · 837 forks · +3 stars in 24 hours · Trending score 1.58
-
10
YOLOW92/pagesift
Local-first web & document ingestion engine in Rust — HTTP/browser capture, layout-aware PDF/Office/OCR parsing, persistent sync, verifiable data graphs, and evidence-grounded AI context.
GitHub · Rust · 284 stars · 37 forks · +1 stars in 24 hours · Trending score 1.13
-
11
Goldziher/basemind
Full AI context and content layer for coding agents over one MCP server — tree-sitter code-map, document RAG, shared memory, multi-agent comms, web crawl, git history + blame. 300+ languages, 10+ agent harnesses, pure Rust.
GitHub · Rust · 124 stars · 24 forks · +1 stars in 24 hours · Trending score 1.05
-
12
K4N3CO/PWNNET-Toolkit
PWN//NET is an ethical bug-hunting and networking toolkit designed for researchers, and enthusiasts. From identifying critical vulnerabilities to uncovering hidden attack surfaces, PWN//NET equips you with 45+ real-world scanners and diagnostic tools.
GitHub · TypeScript · 9 stars · 3 forks · +3 stars in 24 hours · Trending score 0.94
-
13
alphap365/open-news
An extensible Python engine and library for news discovery, aggregation, processing, search, and AI-ready intelligence.
GitHub · Python · 50 stars · 6 forks · +1 stars in 24 hours · Trending score 0.64
-
14
Ayyouboss0011/SherlockMaps
Powerful Google Maps Crawler / Scraper tool with REST API, Docker support & multi-format export
GitHub · JavaScript · 96 stars · 12 forks · 0 stars in 24 hours · Trending score 0.59
-
15
andalabx/ember
Lightweight headless browser for AI agents
GitHub · Python · 10 stars · 2 forks · 0 stars in 24 hours · Trending score 0.56
-
16
netweir/netweir
An ultra-fast, self-healing, undetectable web scraper. Rust does the work; you write Python.
GitHub · Rust · 8 stars · 4 forks · +1 stars in 24 hours · Trending score 0.56
-
17
ScrapeGraphAI/scrapegraph-js
Official JavaScript/TypeScript SDK for the ScrapeGraph AI API. Smart scraping, search, crawling, markdownify, agentic browser automation, scheduled jobs, and structured data extraction — all powered by AI.
GitHub · TypeScript · 8 stars · 10 forks · +1 stars in 24 hours · Trending score 0.56
-
18
wojcikdominik/openspider
The crawler for the agentic era – open-source website crawler & technical SEO auditor for AI agents and humans (macOS, Windows). Rust + Tauri, MCP-native.
GitHub · Rust · 47 stars · 1 forks · 0 stars in 24 hours · Trending score 0.53
-
19
gildas-lormeau/single-file-cli
CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)
GitHub · JavaScript · 1,697 stars · 147 forks · 0 stars in 24 hours · Trending score 0.52
-
20
Adityavardhanjain/WikiCrawl
Interactive Wikipedia knowledge graph - crawl topics, visualize connections, and explore unexpected paths.
GitHub · TypeScript · 29 stars · 3 forks · 0 stars in 24 hours · Trending score 0.49
-
21
realsirjoe/relinos
Organizing the web
GitHub · Python · 56 stars · 5 forks · 0 stars in 24 hours · Trending score 0.46
-
22
MarginaliaSearch/MarginaliaSearch
Internet search engine for text-oriented websites. Indexing the small, old and weird web.
GitHub · Java · 2,241 stars · 68 forks · 0 stars in 24 hours · Trending score 0.46
-
23
scrapfly/scrapfly-scrapers
Scalable Python web scraping scripts for +80 popular domains
GitHub · Python · 1,133 stars · 207 forks · 0 stars in 24 hours · Trending score 0.46
-
24
vincenzo-afk/Intelis-Agent
A self-running research agent that monitors the web on your schedule, verifies findings with LLM analysis, and delivers intelligence reports by email and in-app.
GitHub · TypeScript · 29 stars · 0 forks · +1 stars in 24 hours · Trending score 0.45