data-ingestion repositories
Trending repositories tagged data-ingestion, ranked by trending score. Updated daily.
-
1
datazip-inc/olake-go
OLake Go: open-source replication from databases, Kafka and S3 to Apache Iceberg and Parquet. Sources: Postgres, MySQL, MongoDB, Oracle, MSSQL, DB2, Kafka, S3. Formerly datazip-inc/olake. Table maintenance lives in datazip-inc/olake-fusion.
GitHub · Go · 2,154 stars · 320 forks · +163 stars in 24 hours · Trending score 4.18
-
2
apache/seatunnel
SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.
GitHub · Java · 9,821 stars · 2,474 forks · +4 stars in 24 hours · Trending score 1.76
-
3
oracle-samples/oracle-aidp-samples
Oracle AI Data Platform Workbench Samples
GitHub · Python · 61 stars · 44 forks · +5 stars in 24 hours · Trending score 1.46
-
4
bruin-data/ingestr
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
GitHub · Go · 4,049 stars · 170 forks · +3 stars in 24 hours · Trending score 1.16
-
5
bruin-data/bruin
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
GitHub · Go · 1,816 stars · 100 forks · +1 stars in 24 hours · Trending score 0.91
-
6
CrunchyData/pg_parquet
Copy to/from Parquet in S3, Azure Blob Storage, Google Cloud Storage, http(s) stores, local files or standard inout stream from within PostgreSQL
GitHub · Rust · 696 stars · 40 forks · +2 stars in 24 hours · Trending score 0.81
-
7
apache/paimon
Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
GitHub · Java · 3,454 stars · 1,438 forks · 0 stars in 24 hours · Trending score 0.80
-
8
apache/paimon-rust
Apache Paimon Rust The rust implementation of Apache Paimon.
GitHub · Rust · 215 stars · 115 forks · 0 stars in 24 hours · Trending score 0.43
-
9
cernodatainc/cernodata
Open-source control layer over PDF parsing that picks the right method, verifies output quality, and falls back automatically when results fall short
GitHub · Python · 24 stars · 6 forks · 0 stars in 24 hours · Trending score 0.31
-
10
opencrawling/opencrawling
Reference Java & Spring implementation of the Open Ingestion Standard (OIS). It provides a secure, decoupled, and vendor-neutral enterprise data integration platform leveraging modern Java 25 features, Spring Boot, Spring AI and vector search infrastructure to orchestrate data flows
GitHub · Java · 29 stars · 5 forks · 0 stars in 24 hours · Trending score 0.24
-
11
datris/datris-platform-oss
The Data Control Plane for AI Agents — acquire, validate, and land data over MCP with scoped keys, approval gates, and an audit log. Self-hosted, open source.
GitHub · Scala · 14 stars · 1 forks · 0 stars in 24 hours · Trending score 0.22
-
12
elastic/LogstashUI
A control plane and visual pipeline editor for Logstash
GitHub · Python · 34 stars · 7 forks · 0 stars in 24 hours · Trending score 0.18
-
13
emiliano-go/rypipe
Format- and source-agnostic ingestion framework: a common execution runtime for turning record-oriented data sources into typed columnar data. Rust core, Python bindings; adapters live in separate packages.
GitHub · Rust · 23 stars · 2 forks · 0 stars in 24 hours · Trending score 0.00