data-processing repositories
Trending repositories tagged data-processing, ranked by trending score. Updated daily.
-
1
datajuicer/data-juicer
Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷
GitHub · Python · 7,265 stars · 456 forks · +14 stars in 24 hours · Trending score 2.17
-
2
cocoindex-io/cocoindex
Incremental engine for long horizon agents 🌟 Star if you like it!
GitHub · Rust · 11,923 stars · 930 forks · +13 stars in 24 hours · Trending score 1.99
-
3
johnkerl/miller
Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON
GitHub · Go · 10,069 stars · 261 forks · +4 stars in 24 hours · Trending score 1.51
-
4
NVIDIA-NeMo/Curator
Scalable data pre processing and curation toolkit for LLMs
GitHub · Python · 1,866 stars · 379 forks · +1 stars in 24 hours · Trending score 1.04
-
5
NVIDIA/DALI
A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.
GitHub · C++ · 5,797 stars · 691 forks · +1 stars in 24 hours · Trending score 0.84
-
6
MDSplus/mdsplus
The MDSplus data management system
GitHub · Java · 99 stars · 56 forks · +2 stars in 24 hours · Trending score 0.81
-
7
BrandonML/google-maps-timeline-converter
Google's 2024 format change broke compatibility. This tool unifies old (.json) and new (Timeline.json) data into one clean, usable file.
GitHub · TypeScript · 13 stars · 1 forks · +1 stars in 24 hours · Trending score 0.55
-
8
mech-lang/mech
🦾 Mech is a programming language for building data-driven systems like robots, games, and interfaces. Start here!
GitHub · Rust · 309 stars · 23 forks · 0 stars in 24 hours · Trending score 0.49
-
9
datachain-ai/datachain
The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure
GitHub · Python · 2,832 stars · 180 forks · 0 stars in 24 hours · Trending score 0.41
-
10
pathwaycom/pathway
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
GitHub · Python · 61,981 stars · 1,684 forks · +1 stars in 24 hours · Trending score 0.41
-
11
bytewax/bytewax
Python Stream Processing
GitHub · Python · 2,074 stars · 114 forks · 0 stars in 24 hours · Trending score 0.40
-
12
numaproj/numaflow
Kubernetes-native platform to run massively parallel data/streaming jobs
GitHub · Rust · 2,838 stars · 187 forks · 0 stars in 24 hours · Trending score 0.38
-
13
LiberTEM/LiberTEM
Open pixelated STEM framework
GitHub · Python · 132 stars · 74 forks · 0 stars in 24 hours · Trending score 0.32
-
14
datumbrain/otters
🦦 Pandas-style DataFrame library for Go — fluent API for filtering, grouping, sorting, and statistical analysis with type-safe operations and zero-copy typed slice access.
GitHub · Go · 18 stars · 6 forks · 0 stars in 24 hours · Trending score 0.32
-
15
senbox-org/snap-engine
ESA Earth Observation Toolbox and Java Development Platform
GitHub · Java · 217 stars · 106 forks · 0 stars in 24 hours · Trending score 0.18
-
16
LibreCat/Catmandu
Catmandu - a data processing toolkit
GitHub · Perl · 199 stars · 35 forks · 0 stars in 24 hours · Trending score 0.14
-
17
deverte/awesome-science
A currated list of awesome scientific software, libraries and services.
GitHub · 38 stars · 12 forks · 0 stars in 24 hours · Trending score 0.10
-
18
melihbirim/csvql
Run SQL on CSV files in place. Read-only CSV query engine and MCP server for AI agents. Single Zig binary, no import.
GitHub · Zig · 31 stars · 6 forks · 0 stars in 24 hours · Trending score 0.09
-
19
Puchaczov/Musoq
SQL Runtime without any database
GitHub · C# · 507 stars · 22 forks · −1 stars in 24 hours · Trending score 0.09
-
20
GabrielMarquezMatte/ExcelReader
High-performance, low-allocation Excel (.xlsx, .xlsb, .xls, .csv) reader and writer for .NET 10, with Native AOT support and C++/Rust/Python bindings.
GitHub · C# · 21 stars · 1 forks · 0 stars in 24 hours · Trending score 0.08
-
21
senbox-org/snap-desktop
Desktop GUI for SNAP based on NetBeans Platform
GitHub · Java · 151 stars · 67 forks · 0 stars in 24 hours · Trending score 0.08
-
22
kubeflow/mcp-apache-spark-history-server
MCP Server and CLI for Apache Spark History Server. Debug Spark applications from AI agents, scripts, or the terminal.
GitHub · Python · 208 stars · 85 forks · 0 stars in 24 hours · Trending score 0.07
-
23
polyaxon/haupt
Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon
GitHub · Python · 452 stars · 204 forks · 0 stars in 24 hours · Trending score 0.04
-
24
TangleML/tangle
Tangle is a web app that allows the users to build and run Machine Learning pipelines without having to set up development environment.
GitHub · Python · 516 stars · 41 forks · 0 stars in 24 hours · Trending score 0.04