data-pipelines repositories
Trending repositories tagged data-pipelines, ranked by trending score. Updated daily.
-
1
apache/airflow
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
GitHub · Python · 47,581 stars · 18,355 forks · +44 stars in 24 hours · Trending score 3.32
-
2
dagster-io/dagster
An orchestration platform for the development, production, and observation of data assets.
GitHub · Python · 16,444 stars · 2,427 forks · +9 stars in 24 hours · Trending score 2.36
-
3
elementary-data/dbt-data-reliability
This dbt package captures metadata, artifacts, and test results so you can detect anomalies, monitor data quality, and build metadata tables. It powers Elementary OSS and feeds the wider context layer used by Elementary Cloud’s full Data & AI Control Plane.
GitHub · Python · 531 stars · 161 forks · +5 stars in 24 hours · Trending score 1.60
-
4
feldera/feldera
The Feldera Incremental Computation Engine
GitHub · Rust · 2,181 stars · 181 forks · +1 stars in 24 hours · Trending score 1.18
-
5
ucbepic/docetl
A system for agentic LLM-powered data processing and ETL
GitHub · Python · 4,259 stars · 471 forks · 0 stars in 24 hours · Trending score 1.08
-
6
Unstructured-IO/unstructured
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
GitHub · HTML · 15,759 stars · 1,404 forks · 0 stars in 24 hours · Trending score 1.02
-
7
apache/dolphinscheduler
Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
GitHub · Java · 14,559 stars · 5,122 forks · +2 stars in 24 hours · Trending score 1.00
-
8
bruin-data/bruin
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
GitHub · Go · 1,816 stars · 100 forks · +1 stars in 24 hours · Trending score 0.91
-
9
meltano/meltano
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
GitHub · Python · 2,683 stars · 282 forks · 0 stars in 24 hours · Trending score 0.57
-
10
fabdendev/dagster-mcp
MCP server that wraps the Dagster GraphQL API — monitor and operate runs, assets, schedules, sensors, and backfills from any MCP client
GitHub · Python · 18 stars · 11 forks · +1 stars in 24 hours · Trending score 0.55
-
11
astronomer/agents
AI agent tooling for data engineering workflows.
GitHub · Python · 462 stars · 72 forks · −1 stars in 24 hours · Trending score 0.47
-
12
pathwaycom/pathway
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
GitHub · Python · 61,981 stars · 1,684 forks · +1 stars in 24 hours · Trending score 0.41
-
13
cubert-hyperspectral/cuvis-ai
Cuvis.AI is an open-source, extensible framework for building modular AI pipelines that process, analyze, and deploy hyperspectral video data in real-time applications.
GitHub · Jupyter Notebook · 20 stars · 0 forks · 0 stars in 24 hours · Trending score 0.36
-
14
cernodatainc/cernodata
Open-source control layer over PDF parsing that picks the right method, verifies output quality, and falls back automatically when results fall short
GitHub · Python · 24 stars · 6 forks · 0 stars in 24 hours · Trending score 0.31
-
15
openivm/openivm
Incremental View Maintenance on DuckDB
GitHub · C++ · 71 stars · 7 forks · 0 stars in 24 hours · Trending score 0.30
-
16
linkedin/Hoptimator
Multi-hop declarative data pipelines
GitHub · Java · 128 stars · 19 forks · 0 stars in 24 hours · Trending score 0.25
-
17
KafScale/platform
Kafka-compatible streaming that keeps your data in S3. Stateless brokers, standard Kafka clients, Apache 2.0.
GitHub · Go · 59 stars · 16 forks · 0 stars in 24 hours · Trending score 0.10
-
18
DidactHQ/didact
The open core .NET job orchestrator that we've been missing
GitHub · C# · 162 stars · 3 forks · 0 stars in 24 hours · Trending score 0.10
-
19
eschizoid/kpipe
Composable Kafka consumer library for building modular, testable JVM data pipelines.
GitHub · Java · 58 stars · 7 forks · 0 stars in 24 hours · Trending score 0.10
-
20
debezium/debezium-platform
An opinionated data-centric view of Debezium components. Please log issues at https://github.com/debezium/dbz/issues.
GitHub · TypeScript · 55 stars · 45 forks · 0 stars in 24 hours · Trending score 0.09
-
21
databricks-solutions/lakeflow_framework
Metadata-driven framework for Databricks Spark Declarative Pipelines. Config-driven, pattern based approach to batch & streaming across the medallion architecture. Deploys via Declarative Automation Bundles. Built for simplicity, extensibility, and alignment with the Databricks product roadmap.
GitHub · Python · 30 stars · 20 forks · 0 stars in 24 hours · Trending score 0.09
-
22
akshaysharma1088/dataranger
Open-source AI agent that patrols your data, fixes only what it can prove against the source, and sends the rest to a person with the evidence. Works with Snowflake, dbt and DuckDB, runs as a GitHub Action, and speaks MCP.
GitHub · Python · 5 stars · 10 forks · 0 stars in 24 hours · Trending score 0.07
-
23
smart-data-lake/smart-data-lake
Smart Automation Tool for building modern Data Lakes and Data Pipelines
GitHub · Scala · 130 stars · 28 forks · 0 stars in 24 hours · Trending score 0.05
-
24
CogStack/CogStack-NiFi
Building data processing pipelines for documents processing with NLP using Apache NiFi and related services
GitHub · Python · 61 stars · 20 forks · 0 stars in 24 hours · Trending score 0.05