data-processing repositories

Trending repositories tagged data-processing, ranked by trending score. Updated daily.

  1. 1
    datajuicer/data-juicer

    Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷

    GitHub · Python · 7,265 stars · 456 forks · +14 stars in 24 hours · Trending score 2.17

  2. 2
    cocoindex-io/cocoindex

    Incremental engine for long horizon agents 🌟 Star if you like it!

    GitHub · Rust · 11,923 stars · 930 forks · +13 stars in 24 hours · Trending score 1.99

  3. 3
    johnkerl/miller

    Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON

    GitHub · Go · 10,069 stars · 261 forks · +4 stars in 24 hours · Trending score 1.51

  4. 4
    NVIDIA-NeMo/Curator

    Scalable data pre processing and curation toolkit for LLMs

    GitHub · Python · 1,866 stars · 379 forks · +1 stars in 24 hours · Trending score 1.04

  5. 5
    NVIDIA/DALI

    A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.

    GitHub · C++ · 5,797 stars · 691 forks · +1 stars in 24 hours · Trending score 0.84

  6. 6
    MDSplus/mdsplus

    The MDSplus data management system

    GitHub · Java · 99 stars · 56 forks · +2 stars in 24 hours · Trending score 0.81

  7. 7
    BrandonML/google-maps-timeline-converter

    Google's 2024 format change broke compatibility. This tool unifies old (.json) and new (Timeline.json) data into one clean, usable file.

    GitHub · TypeScript · 13 stars · 1 forks · +1 stars in 24 hours · Trending score 0.55

  8. 8
    mech-lang/mech

    🦾 Mech is a programming language for building data-driven systems like robots, games, and interfaces. Start here!

    GitHub · Rust · 309 stars · 23 forks · 0 stars in 24 hours · Trending score 0.49

  9. 9
    datachain-ai/datachain

    The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure

    GitHub · Python · 2,832 stars · 180 forks · 0 stars in 24 hours · Trending score 0.41

  10. 10
    pathwaycom/pathway

    Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

    GitHub · Python · 61,981 stars · 1,684 forks · +1 stars in 24 hours · Trending score 0.41

  11. 11
    bytewax/bytewax

    Python Stream Processing

    GitHub · Python · 2,074 stars · 114 forks · 0 stars in 24 hours · Trending score 0.40

  12. 12
    numaproj/numaflow

    Kubernetes-native platform to run massively parallel data/streaming jobs

    GitHub · Rust · 2,838 stars · 187 forks · 0 stars in 24 hours · Trending score 0.38

  13. 13
    LiberTEM/LiberTEM

    Open pixelated STEM framework

    GitHub · Python · 132 stars · 74 forks · 0 stars in 24 hours · Trending score 0.32

  14. 14
    datumbrain/otters

    🦦 Pandas-style DataFrame library for Go — fluent API for filtering, grouping, sorting, and statistical analysis with type-safe operations and zero-copy typed slice access.

    GitHub · Go · 18 stars · 6 forks · 0 stars in 24 hours · Trending score 0.32

  15. 15
    senbox-org/snap-engine

    ESA Earth Observation Toolbox and Java Development Platform

    GitHub · Java · 217 stars · 106 forks · 0 stars in 24 hours · Trending score 0.18

  16. 16
    LibreCat/Catmandu

    Catmandu - a data processing toolkit

    GitHub · Perl · 199 stars · 35 forks · 0 stars in 24 hours · Trending score 0.14

  17. 17
    deverte/awesome-science

    A currated list of awesome scientific software, libraries and services.

    GitHub · 38 stars · 12 forks · 0 stars in 24 hours · Trending score 0.10

  18. 18
    melihbirim/csvql

    Run SQL on CSV files in place. Read-only CSV query engine and MCP server for AI agents. Single Zig binary, no import.

    GitHub · Zig · 31 stars · 6 forks · 0 stars in 24 hours · Trending score 0.09

  19. 19
    Puchaczov/Musoq

    SQL Runtime without any database

    GitHub · C# · 507 stars · 22 forks · −1 stars in 24 hours · Trending score 0.09

  20. 20
    GabrielMarquezMatte/ExcelReader

    High-performance, low-allocation Excel (.xlsx, .xlsb, .xls, .csv) reader and writer for .NET 10, with Native AOT support and C++/Rust/Python bindings.

    GitHub · C# · 21 stars · 1 forks · 0 stars in 24 hours · Trending score 0.08

  21. 21
    senbox-org/snap-desktop

    Desktop GUI for SNAP based on NetBeans Platform

    GitHub · Java · 151 stars · 67 forks · 0 stars in 24 hours · Trending score 0.08

  22. 22
    kubeflow/mcp-apache-spark-history-server

    MCP Server and CLI for Apache Spark History Server. Debug Spark applications from AI agents, scripts, or the terminal.

    GitHub · Python · 208 stars · 85 forks · 0 stars in 24 hours · Trending score 0.07

  23. 23
    polyaxon/haupt

    Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon

    GitHub · Python · 452 stars · 204 forks · 0 stars in 24 hours · Trending score 0.04

  24. 24
    TangleML/tangle

    Tangle is a web app that allows the users to build and run Machine Learning pipelines without having to set up development environment.

    GitHub · Python · 516 stars · 41 forks · 0 stars in 24 hours · Trending score 0.04