warc repositories

Trending repositories tagged warc, ranked by trending score. Updated daily.

  1. 1
    ArchiveBox/ArchiveBox

    馃梼 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

    GitHub 路 Python 路 29,079 stars 路 1,635 forks 路 +20 stars in 24 hours 路 Trending score 2.27

  2. 2
    webrecorder/warcio

    Streaming WARC/ARC library for fast web archive IO

    GitHub 路 Python 路 486 stars 路 82 forks 路 +4 stars in 24 hours 路 Trending score 1.32

  3. 3
    webrecorder/replayweb.page

    Serverless replay of web archives directly in the browser

    GitHub 路 TypeScript 路 1,016 stars 路 96 forks 路 0 stars in 24 hours 路 Trending score 0.90

  4. 4
    commoncrawl/news-crawl

    News crawling with StormCrawler - stores content as WARC

    GitHub 路 Java 路 384 stars 路 44 forks 路 +1 stars in 24 hours 路 Trending score 0.55

  5. 5
    ArchiveTeam/wget-lua

    Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

    GitHub 路 C 路 140 stars 路 19 forks 路 +1 stars in 24 hours 路 Trending score 0.45

  6. 6
    webrecorder/browsertrix

    Browsertrix is the hosted, high-fidelity, browser-based crawling service from Webrecorder designed to make web archiving easier and more accessible for all!

    GitHub 路 TypeScript 路 502 stars 路 83 forks 路 0 stars in 24 hours 路 Trending score 0.36

  7. 7
    openzim/zimit-frontend

    Zimit Public Web UI

    GitHub 路 Python 路 25 stars 路 20 forks 路 0 stars in 24 hours 路 Trending score 0.19

  8. 8
    internetarchive/gowarc

    Read and write WARC files in Go

    GitHub 路 Go 路 56 stars 路 12 forks 路 0 stars in 24 hours 路 Trending score 0.06

  9. 9
    webrecorder/browsertrix-crawler

    Run a high-fidelity browser-based web archiving crawler in a single Docker container

    GitHub 路 TypeScript 路 1,190 stars 路 155 forks 路 0 stars in 24 hours 路 Trending score 0.02

  10. 10
    centic9/CommonCrawlDocumentDownload

    A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types. This is used for mass-testing of frameworks like Apache POI and Apache Tika

    GitHub 路 Java 路 74 stars 路 16 forks 路 0 stars in 24 hours 路 Trending score 0.00

  11. 11
    commoncrawl/whirlwind-python

    A whirlwind tour of Common Crawl's data using Python

    GitHub 路 Python 路 47 stars 路 10 forks 路 0 stars in 24 hours 路 Trending score 0.00

  12. 12
    ruarxive/awesome-digital-preservation

    Awesome list dedicated to digital and data preservation tools, sources, services and so on.

    GitHub 路 39 stars 路 12 forks 路 0 stars in 24 hours 路 Trending score 0.00