data-pipeline repositories
Trending repositories tagged data-pipeline, ranked by trending score. Updated daily.
-
1
datazip-inc/olake-go
OLake Go: open-source replication from databases, Kafka and S3 to Apache Iceberg and Parquet. Sources: Postgres, MySQL, MongoDB, Oracle, MSSQL, DB2, Kafka, S3. Formerly datazip-inc/olake. Table maintenance lives in datazip-inc/olake-fusion.
GitHub · Go · 2,130 stars · 318 forks · +163 stars in 24 hours · Trending score 4.18
-
2
qiantongtech/qData
qData is a lightweight, feature-complete, AI-native open-source data platform. It covers data integration, development, modeling, governance, assets and API services, with self-hosted deployment, flexible scaling and commercial use.
GitHub · Java · 944 stars · 140 forks · +36 stars in 24 hours · Trending score 2.69
-
3
dagucloud/dagu
Self-hostable workflow orchestrator for teams whose main work isn't orchestration. Declarative YAML over your scripts, SSH commands, containers, etc; keep workflows separate from business logic. One binary, no database, runs on limited H/W resources. Alternative to Airflow / Cron / Job Scheduler.
GitHub · Go · 4,639 stars · 408 forks · +12 stars in 24 hours · Trending score 2.27
-
4
datajuicer/data-juicer
Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷
GitHub · Python · 7,265 stars · 456 forks · +14 stars in 24 hours · Trending score 2.17
-
5
galaxy-io/filament
Pluggable data replication with checkpointing, batching, and integrity events
GitHub · Go · 336 stars · 47 forks · +8 stars in 24 hours · Trending score 2.16
-
6
superlinked/sie
Open-source inference server and production cluster for all the models your agent needs.
GitHub · Python · 3,533 stars · 339 forks · +4 stars in 24 hours · Trending score 1.69
-
7
debezium/debezium
Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.
GitHub · Java · 13,365 stars · 3,134 forks · +3 stars in 24 hours · Trending score 1.64
-
8
airbytehq/airbyte
Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
GitHub · Python · 22,411 stars · 5,550 forks · 0 stars in 24 hours · Trending score 1.47
-
9
yuyuejia/datay
DataY 是一套轻量、高性能的数据平台,一个 JAR、自研流批一体处理引擎,自研分布式调度框架,在 2 核 4G 的机器上 就能跑通 数据集成 → 数据建模 → 数据开发 → 任务调度 → 数据应用 的完整链路。
GitHub · Java · 50 stars · 23 forks · +3 stars in 24 hours · Trending score 1.41
-
10
bruin-data/ingestr
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
GitHub · Go · 4,049 stars · 170 forks · +3 stars in 24 hours · Trending score 1.16
-
11
Unknown-333/awesome-data-engineering-skills
36 data-engineering skills for AI coding agents — dbt, Airflow, Dagster, Prefect, Spark, Snowflake, BigQuery, Databricks, Kafka, Iceberg. Portable SKILL.md Agent Skills (the format Anthropic launched as Claude Skills), with idempotency, backfill, and data-quality know-how. Runs in Claude Code, Cursor, Codex, Copilot, Gemini CLI and more.
GitHub · Python · 27 stars · 1 forks · +4 stars in 24 hours · Trending score 1.16
-
12
ultralytics/stars
Daily Ultralytics analytics for GitHub, PyPI, Google Analytics, Reddit, and Platform metrics, published as static JSON with historical star tracking.
GitHub · Python · 102 stars · 6 forks · +2 stars in 24 hours · Trending score 0.91
-
13
vaquarkhan/data-engineering-agent-skills
Production-grade Agent Skills for data engineering AI agents: 73 workflows, platform presets, safe backfill/replay, Kafka & Spark reliability, MCP observability, and VS Code/JetBrains installers.
GitHub · Python · 65 stars · 9 forks · −1 stars in 24 hours · Trending score 0.80
-
14
estuary/flow
🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊
GitHub · Rust · 992 stars · 101 forks · +1 stars in 24 hours · Trending score 0.73
-
15
Hebbian-Robotics/hflow
SDK for robotics teams to verify the quality of their data used for AI model training.
GitHub · Python · 310 stars · 171 forks · 0 stars in 24 hours · Trending score 0.67
-
16
iflytek/dolphin-mcp-pilot
Production-ready MCP server for Apache DolphinScheduler (小海豚) — lets AI agents build, schedule, run and recover workflows
GitHub · Python · 24 stars · 22 forks · 0 stars in 24 hours · Trending score 0.65
-
17
apache/shardingsphere
Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.
GitHub · Java · 20,832 stars · 6,911 forks · 0 stars in 24 hours · Trending score 0.62
-
18
PkLavc/PkLavc.github.io
Personal portfolio and engineering website for Patrick Araujo, focused on backend systems, AI, automation, API integrations, and technical projects.
GitHub · HTML · 34 stars · 0 forks · +1 stars in 24 hours · Trending score 0.55
-
19
mloda-ai/mloda
mloda.ai - Open Data Access for AI and ML. Plugin-based. Traceable. Framework-agnostic.
GitHub · Python · 101 stars · 106 forks · +1 stars in 24 hours · Trending score 0.55
-
20
clidey/whodb
Where data access meets operational intelligence
GitHub · Go · 5,061 stars · 243 forks · 0 stars in 24 hours · Trending score 0.47
-
21
simonesiega/european-tech-opportunities-2027
Your open-source hub for 2027 tech internships and New Grad opportunities across Europe.
GitHub · Python · 55 stars · 0 forks · 0 stars in 24 hours · Trending score 0.46
-
22
ozkary/data-engineering-mta-turnstile
Data Engineering - Metropolitan Transportation Authority (MTA) Subway Data Analysis
GitHub · Jupyter Notebook · 34 stars · 7 forks · +1 stars in 24 hours · Trending score 0.45
-
23
rudderlabs/rudder-server
Privacy and Security focused Segment-alternative, in Golang and React
GitHub · Go · 4,520 stars · 84 forks · 0 stars in 24 hours · Trending score 0.40
-
24
analyticsdurgesh/DBT-Project-Pipeline
Public Snowflake and dbt pipeline transforming Blinkit retail sales data into staging and mart analytics models.
GitHub · Python · 25 stars · 4 forks · 0 stars in 24 hours · Trending score 0.36