scraping repositories
Trending repositories tagged scraping, ranked by trending score. Updated daily.
-
1
firecrawl/firecrawl
Knowledge infrastructure to make your agents smarter. Search, scrape and 550+ data providers through one API.
GitHub · TypeScript · 204,946 stars · 10,743 forks · +1,470 stars in 24 hours · Trending score 5.69
-
2
D4Vinci/Scrapling
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
GitHub · Python · 104,598 stars · 10,941 forks · +468 stars in 24 hours · Trending score 5.28
-
3
soxoj/maigret
🕵️♂️ Collect a dossier on a person by username from 6K websites
GitHub · Python · 41,084 stars · 3,324 forks · +193 stars in 24 hours · Trending score 4.41
-
4
scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
GitHub · Python · 65,511 stars · 12,162 forks · +47 stars in 24 hours · Trending score 3.40
-
5
jo-inc/camofox-browser
Stealth headless browser for AI agents — bypass Cloudflare, bot detection, and anti-scraping. Drop-in Puppeteer/Playwright replacement.
GitHub · JavaScript · 12,424 stars · 1,270 forks · +54 stars in 24 hours · Trending score 3.34
-
6
apify/crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
GitHub · TypeScript · 26,505 stars · 1,774 forks · +40 stars in 24 hours · Trending score 3.23
-
7
tiliondev/fortress
Stealth Chromium engine that stops scrapers and browser agents from getting blocked, with one line of code change.
GitHub · Python · 810 stars · 55 forks · +86 stars in 24 hours · Trending score 3.22
-
8
spider-rs/spider
Foundational low latency web data collecting in Rust
GitHub · Rust · 2,855 stars · 331 forks · +24 stars in 24 hours · Trending score 2.48
-
9
oss-invisible/playwright
invisible playwright: Playwright that anti-bots cannot see, so no captchas: same API, anti-detect stealth headless Firefox, undetected fingerprint, bypass bot detection, Python web scraping and browser automation.
GitHub · Python · 2,610 stars · 279 forks · +16 stars in 24 hours · Trending score 2.26
-
10
soxoj/socid-extractor
⛏️ The extraction engine behind Maigret: turn any profile URL into a structured OSINT record across 150+ sites
GitHub · Python · 1,141 stars · 133 forks · +15 stars in 24 hours · Trending score 2.15
-
11
proxifly/free-proxy-list
🚀 Free HTTP, SOCKS4, & SOCKS5 proxy list * Updated every 5 minutes * and rotating proxy API (100+ countries)
GitHub · 7,069 stars · 804 forks · +11 stars in 24 hours · Trending score 1.91
-
12
apify/crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.
GitHub · Python · 9,680 stars · 837 forks · +3 stars in 24 hours · Trending score 1.59
-
13
ymstar/dyhub
抖音直播弹幕中台 (Douyin Live Danmaku Hub):双采集内核(浏览器+CDP / 轻量直连)、多房间并发、WS/SSE/Webhook 三通道分发、可视化控制台、Docker 一键部署
GitHub · TypeScript · 61 stars · 20 forks · +5 stars in 24 hours · Trending score 1.52
-
14
symfony/panther
A browser testing and web crawling library for PHP and Symfony
GitHub · PHP · 3,074 stars · 237 forks · +6 stars in 24 hours · Trending score 1.46
-
15
h4ckf0r0day/awesome-ai-web-scraping
A curated list of AI-powered web scraping tools, LLM-friendly crawlers, MCP servers, and infrastructure for turning the web into data.
GitHub · 131 stars · 71 forks · 0 stars in 24 hours · Trending score 1.41
-
16
jikan-me/jikan-rest
The REST API for Jikan
GitHub · PHP · 571 stars · 300 forks · +5 stars in 24 hours · Trending score 1.26
-
17
adbar/trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
GitHub · Python · 7,138 stars · 510 forks · 0 stars in 24 hours · Trending score 1.14
-
18
iiab/iiab
Internet-in-a-Box - Build your own LIBRARY OF ALEXANDRIA with a Raspberry Pi !
GitHub · Jinja · 2,040 stars · 154 forks · 0 stars in 24 hours · Trending score 1.07
-
19
Jieyab89/OSINT-Cheat-sheet
OSINT cheat sheet, list OSINT tools, wiki, dataset, article, book , red team OSINT for hackers and OSINT tips and OSINT branch. This repository will grow every time will research, there is a research, science and technology, tutorial. Please use it wisely.
GitHub · HTML · 2,327 stars · 400 forks · 0 stars in 24 hours · Trending score 1.06
-
20
yaroslav2901/OE_OUTAGE_DATA
Репозиторій містить JSON-дані про відключення електроенергії та згенеровані зображення розкладів для кожної групи та загальне зображення для всіх груп.
GitHub · 40 stars · 2 forks · +3 stars in 24 hours · Trending score 1.05
-
21
alex-on-ai/WebReaper
AI-native web scraper. Single binary with a bundled Claude Code skill. MIT-licensed alternative to Firecrawl.
GitHub · C# · 153 stars · 33 forks · +4 stars in 24 hours · Trending score 1.05
-
22
sqdshguy/wreq-js
HTTP client for Node.js with browser TLS fingerprint impersonation
GitHub · TypeScript · 460 stars · 20 forks · 0 stars in 24 hours · Trending score 1.03
-
23
TulipFarm/huntsman
Self-hosted web scraper that turns any web page into clean Markdown. Firecrawl-compatible API, powered by Lightpanda.
GitHub · Go · 15 stars · 1 forks · +2 stars in 24 hours · Trending score 0.96
-
24
any4ai/AnyCrawl
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
GitHub · TypeScript · 3,528 stars · 372 forks · 0 stars in 24 hours · Trending score 0.96