spider repositories
Trending repositories tagged spider, ranked by trending score. Updated daily.
-
1
spider-rs/spider
Foundational low latency web data collecting in Rust
GitHub · Rust · 2,855 stars · 331 forks · +23 stars in 24 hours · Trending score 2.47
-
2
cv-cat/catbus
一个命令操作小红书、抖音、TikTok、B 站、快手、微博、闲鱼、淘宝、京东、X:统一的命令与 JSON 输出,给人和 AI Agent 用
GitHub · TypeScript · 638 stars · 198 forks · +7 stars in 24 hours · Trending score 2.03
-
3
wujunwei928/parse-video-py
Python短视频去水印爬虫:抖音,皮皮虾,火山,微视,最右,快手,全民小视频,皮皮搞笑,西瓜视频,虎牙,梨视频,acfun,好看视频...
GitHub · Python · 720 stars · 190 forks · +11 stars in 24 hours · Trending score 1.92
-
4
PhiFever/AfdianToMarkdown
爱发电爬虫(afdian.com)
GitHub · Go · 81 stars · 16 forks · +4 stars in 24 hours · Trending score 1.29
-
5
NaiboWang/EasySpider
A visual no-code/code-free web crawler/spider易采集:一个可视化浏览器自动化测试/数据采集/网页爬虫软件,可以无代码图形化的设计和执行爬虫任务。别名:ServiceWrapper面向Web应用的智能化服务封装系统。
GitHub · JavaScript · 44,801 stars · 5,393 forks · +3 stars in 24 hours · Trending score 1.23
-
6
CharlesPikachu/freeproxy
FreeProxy: Collecting free proxies from internet. (全球海量高质量免费代理,支持爬取数十个免费代理分享源,支持自定义规则代理筛选,爬虫与数据分析必备,每日更新海量免费代理。)
GitHub · Python · 592 stars · 103 forks · +2 stars in 24 hours · Trending score 1.19
-
7
zorlan/skycaiji
蓝天采集器是一款开源免费的爬虫系统,仅需点选编辑规则即可采集数据,可运行在本地、虚拟主机或云服务器中,几乎能采集所有类型的网页,无缝对接各类CMS建站程序,免登录实时发布数据,全自动无需人工干预!是网页大数据采集软件中完全跨平台的云端爬虫系统
GitHub · PHP · 2,098 stars · 605 forks · +3 stars in 24 hours · Trending score 0.90
-
8
signal18/replication-manager
Signal 18 repman - Replication Manager for MySQL / MariaDB / Percona Server
GitHub · Go · 747 stars · 177 forks · +1 stars in 24 hours · Trending score 0.63
-
9
niubihu1/tvbox
🐯 牛虎 ┃ TVBox 旗舰聚合接口 · 4K原画秒播 · 纯净零广告 · 2000+高清电视直播(含EPG台标) · 影音壳单仓与多仓精选 · 双核爬虫驱动 · 大数据共识优选 · 每日自动巡检保活与去死链
GitHub · JavaScript · 237 stars · 53 forks · 0 stars in 24 hours · Trending score 0.62
-
10
bda-research/node-crawler
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
GitHub · TypeScript · 6,797 stars · 863 forks · +1 stars in 24 hours · Trending score 0.55
-
11
ndgigliotti/shopify-spy
Python library and CLI for scraping products, images and collections from Shopify and WooCommerce stores.
GitHub · Python · 124 stars · 54 forks · 0 stars in 24 hours · Trending score 0.48
-
12
scrapfly/scrapfly-scrapers
Scalable Python web scraping scripts for +80 popular domains
GitHub · Python · 1,131 stars · 207 forks · 0 stars in 24 hours · Trending score 0.46
-
13
scrapfly/Antibot-Detector
Real-time detection of anti-bot systems, captchas & fingerprinting techniques. Identifies Cloudflare, Akamai, DataDome, reCAPTCHA, hCaptcha, Shape Security & more with confidence scoring and advanced capture tools.
GitHub · JavaScript · 632 stars · 74 forks · 0 stars in 24 hours · Trending score 0.45
-
14
DedSecInside/TorBot
Dark Web OSINT Tool
GitHub · Python · 5,197 stars · 780 forks · 0 stars in 24 hours · Trending score 0.40
-
15
alltheplaces/alltheplaces
A set of spiders and scrapers to extract location information from places that post their location on the internet.
GitHub · Python · 852 stars · 311 forks · 0 stars in 24 hours · Trending score 0.38
-
16
BlessedRebuS/Krawl
Krawl is a customizable, lightweight, cloud-native web deception server and anti-crawler that creates fake web applications with low-hanging vulnerabilities using realistic, randomly generated decoy data and AI-generated HTML templates.
GitHub · Python · 841 stars · 77 forks · 0 stars in 24 hours · Trending score 0.33
-
17
polyrabbit/hacker-news-digest
:newspaper: Let LLM Summarize Hacker News for You
GitHub · Python · 754 stars · 94 forks · 0 stars in 24 hours · Trending score 0.21
-
18
spiritLHLS/Hang-up-items
问卷调查项目,云服务器推荐,挂机项目,免费代理,各种脚本收集。欢迎右上角点铃铛及时收取更新信息。(不要fork,低调) Questionnaire project, cloud server recommendation, hanging project, free proxy, various script collection. Welcome to the upper right corner of the point bell to receive timely updates. (Do not fork, low profile)
GitHub · 648 stars · 64 forks · 0 stars in 24 hours · Trending score 0.18
-
19
qnxg/hnu_query
湖南大学校内系统查询库,将湖南大学的校内系统(如个人门户,教务系统等)的接口进行包装,提供一套获取结构化数据的能力。
GitHub · Rust · 21 stars · 6 forks · 0 stars in 24 hours · Trending score 0.14
-
20
postmodern/spidr
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
GitHub · Ruby · 837 stars · 105 forks · 0 stars in 24 hours · Trending score 0.14
-
21
yutao8/starred
GitHub Star 收藏资源库,自动同步个人收藏项目。收录 2000+ 个精选开源项目,覆盖开发框架、组件、SDK、模板、API、脚本、工具及各类实用资源,帮助开发者快速发现优质项目。
GitHub · PHP · 176 stars · 22 forks · 0 stars in 24 hours · Trending score 0.10
-
22
Norconex/crawler
Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or filesystem to various data repositories such as search engines.
GitHub · Java · 205 stars · 71 forks · 0 stars in 24 hours · Trending score 0.09
-
23
elixir-crawly/crawly
Crawly, a high-level web crawling & scraping framework for Elixir.
GitHub · Elixir · 1,129 stars · 127 forks · −2 stars in 24 hours · Trending score 0.09
-
24
dirtyfilthy/freshonions-torscraper
Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion
GitHub · Python · 544 stars · 142 forks · 0 stars in 24 hours · Trending score 0.09