warc repositories
Trending repositories tagged warc, ranked by trending score. Updated daily.
-
1
ArchiveBox/ArchiveBox
馃梼 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...
GitHub 路 Python 路 29,079 stars 路 1,635 forks 路 +20 stars in 24 hours 路 Trending score 2.27
-
2
webrecorder/warcio
Streaming WARC/ARC library for fast web archive IO
GitHub 路 Python 路 486 stars 路 82 forks 路 +4 stars in 24 hours 路 Trending score 1.32
-
3
webrecorder/replayweb.page
Serverless replay of web archives directly in the browser
GitHub 路 TypeScript 路 1,016 stars 路 96 forks 路 0 stars in 24 hours 路 Trending score 0.90
-
4
commoncrawl/news-crawl
News crawling with StormCrawler - stores content as WARC
GitHub 路 Java 路 384 stars 路 44 forks 路 +1 stars in 24 hours 路 Trending score 0.55
-
5
ArchiveTeam/wget-lua
Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.
GitHub 路 C 路 140 stars 路 19 forks 路 +1 stars in 24 hours 路 Trending score 0.45
-
6
webrecorder/browsertrix
Browsertrix is the hosted, high-fidelity, browser-based crawling service from Webrecorder designed to make web archiving easier and more accessible for all!
GitHub 路 TypeScript 路 502 stars 路 83 forks 路 0 stars in 24 hours 路 Trending score 0.36
-
7
openzim/zimit-frontend
Zimit Public Web UI
GitHub 路 Python 路 25 stars 路 20 forks 路 0 stars in 24 hours 路 Trending score 0.19
-
8
internetarchive/gowarc
Read and write WARC files in Go
GitHub 路 Go 路 56 stars 路 12 forks 路 0 stars in 24 hours 路 Trending score 0.06
-
9
webrecorder/browsertrix-crawler
Run a high-fidelity browser-based web archiving crawler in a single Docker container
GitHub 路 TypeScript 路 1,190 stars 路 155 forks 路 0 stars in 24 hours 路 Trending score 0.02
-
10
centic9/CommonCrawlDocumentDownload
A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types. This is used for mass-testing of frameworks like Apache POI and Apache Tika
GitHub 路 Java 路 74 stars 路 16 forks 路 0 stars in 24 hours 路 Trending score 0.00
-
11
commoncrawl/whirlwind-python
A whirlwind tour of Common Crawl's data using Python
GitHub 路 Python 路 47 stars 路 10 forks 路 0 stars in 24 hours 路 Trending score 0.00
-
12
ruarxive/awesome-digital-preservation
Awesome list dedicated to digital and data preservation tools, sources, services and so on.
GitHub 路 39 stars 路 12 forks 路 0 stars in 24 hours 路 Trending score 0.00