data-pipelines repositories

Trending repositories tagged data-pipelines, ranked by trending score. Updated daily.

  1. 1
    apache/airflow

    Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

    GitHub · Python · 47,581 stars · 18,355 forks · +44 stars in 24 hours · Trending score 3.32

  2. 2
    dagster-io/dagster

    An orchestration platform for the development, production, and observation of data assets.

    GitHub · Python · 16,444 stars · 2,427 forks · +9 stars in 24 hours · Trending score 2.36

  3. 3
    elementary-data/dbt-data-reliability

    This dbt package captures metadata, artifacts, and test results so you can detect anomalies, monitor data quality, and build metadata tables. It powers Elementary OSS and feeds the wider context layer used by Elementary Cloud’s full Data & AI Control Plane.

    GitHub · Python · 531 stars · 161 forks · +5 stars in 24 hours · Trending score 1.60

  4. 4
    feldera/feldera

    The Feldera Incremental Computation Engine

    GitHub · Rust · 2,181 stars · 181 forks · +1 stars in 24 hours · Trending score 1.18

  5. 5
    ucbepic/docetl

    A system for agentic LLM-powered data processing and ETL

    GitHub · Python · 4,259 stars · 471 forks · 0 stars in 24 hours · Trending score 1.08

  6. 6
    Unstructured-IO/unstructured

    Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.

    GitHub · HTML · 15,759 stars · 1,404 forks · 0 stars in 24 hours · Trending score 1.02

  7. 7
    apache/dolphinscheduler

    Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

    GitHub · Java · 14,559 stars · 5,122 forks · +2 stars in 24 hours · Trending score 1.00

  8. 8
    bruin-data/bruin

    Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

    GitHub · Go · 1,816 stars · 100 forks · +1 stars in 24 hours · Trending score 0.91

  9. 9
    meltano/meltano

    Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.

    GitHub · Python · 2,683 stars · 282 forks · 0 stars in 24 hours · Trending score 0.57

  10. 10
    fabdendev/dagster-mcp

    MCP server that wraps the Dagster GraphQL API — monitor and operate runs, assets, schedules, sensors, and backfills from any MCP client

    GitHub · Python · 18 stars · 11 forks · +1 stars in 24 hours · Trending score 0.55

  11. 11
    astronomer/agents

    AI agent tooling for data engineering workflows.

    GitHub · Python · 462 stars · 72 forks · −1 stars in 24 hours · Trending score 0.47

  12. 12
    pathwaycom/pathway

    Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

    GitHub · Python · 61,981 stars · 1,684 forks · +1 stars in 24 hours · Trending score 0.41

  13. 13
    cubert-hyperspectral/cuvis-ai

    Cuvis.AI is an open-source, extensible framework for building modular AI pipelines that process, analyze, and deploy hyperspectral video data in real-time applications.

    GitHub · Jupyter Notebook · 20 stars · 0 forks · 0 stars in 24 hours · Trending score 0.36

  14. 14
    cernodatainc/cernodata

    Open-source control layer over PDF parsing that picks the right method, verifies output quality, and falls back automatically when results fall short

    GitHub · Python · 24 stars · 6 forks · 0 stars in 24 hours · Trending score 0.31

  15. 15
    openivm/openivm

    Incremental View Maintenance on DuckDB

    GitHub · C++ · 71 stars · 7 forks · 0 stars in 24 hours · Trending score 0.30

  16. 16
    linkedin/Hoptimator

    Multi-hop declarative data pipelines

    GitHub · Java · 128 stars · 19 forks · 0 stars in 24 hours · Trending score 0.25

  17. 17
    KafScale/platform

    Kafka-compatible streaming that keeps your data in S3. Stateless brokers, standard Kafka clients, Apache 2.0.

    GitHub · Go · 59 stars · 16 forks · 0 stars in 24 hours · Trending score 0.10

  18. 18
    DidactHQ/didact

    The open core .NET job orchestrator that we've been missing

    GitHub · C# · 162 stars · 3 forks · 0 stars in 24 hours · Trending score 0.10

  19. 19
    eschizoid/kpipe

    Composable Kafka consumer library for building modular, testable JVM data pipelines.

    GitHub · Java · 58 stars · 7 forks · 0 stars in 24 hours · Trending score 0.10

  20. 20
    debezium/debezium-platform

    An opinionated data-centric view of Debezium components. Please log issues at https://github.com/debezium/dbz/issues.

    GitHub · TypeScript · 55 stars · 45 forks · 0 stars in 24 hours · Trending score 0.09

  21. 21
    databricks-solutions/lakeflow_framework

    Metadata-driven framework for Databricks Spark Declarative Pipelines. Config-driven, pattern based approach to batch & streaming across the medallion architecture. Deploys via Declarative Automation Bundles. Built for simplicity, extensibility, and alignment with the Databricks product roadmap.

    GitHub · Python · 30 stars · 20 forks · 0 stars in 24 hours · Trending score 0.09

  22. 22
    akshaysharma1088/dataranger

    Open-source AI agent that patrols your data, fixes only what it can prove against the source, and sends the rest to a person with the evidence. Works with Snowflake, dbt and DuckDB, runs as a GitHub Action, and speaks MCP.

    GitHub · Python · 5 stars · 10 forks · 0 stars in 24 hours · Trending score 0.07

  23. 23
    smart-data-lake/smart-data-lake

    Smart Automation Tool for building modern Data Lakes and Data Pipelines

    GitHub · Scala · 130 stars · 28 forks · 0 stars in 24 hours · Trending score 0.05

  24. 24
    CogStack/CogStack-NiFi

    Building data processing pipelines for documents processing with NLP using Apache NiFi and related services

    GitHub · Python · 61 stars · 20 forks · 0 stars in 24 hours · Trending score 0.05