pyspark repositories
Trending repositories tagged pyspark, ranked by trending score. Updated daily.
-
1
kengio/databricks-certification-study-guide
Open-source community study guide for all six Databricks certifications (Data Engineer Associate / Professional, Data Analyst Associate, ML Associate / Professional, GenAI Engineer Associate). Aligned to the 2025-2026 official exam guides. Obsidian-flavoured Markdown; PRs welcome.
GitHub · JavaScript · 78 stars · 36 forks · +26 stars in 24 hours · Trending score 2.58
-
2
pavanbadempet/AI-Healthcare-System
AI Healthcare System & Clinical Data Platform. Combines a Databricks Medallion Lakehouse with TabICLv2 tabular foundation models, calibrated tree ensembles (CatBoost, XGBoost), multi-agent RAG, and clinical decision support portal.
GitHub · Python · 75 stars · 19 forks · +9 stars in 24 hours · Trending score 1.71
-
3
zorost/AI-Engineering-Lab
A free, self-paced 24-week AI engineering course: Python, machine learning, LLMs, RAG, fine-tuning, agents and MCP, Azure and Vertex and Bedrock, and Databricks. 43 runnable notebooks, one continuous case study. MIT licensed, no signup. By Zorost Intelligence AI Lab.
GitHub · Jupyter Notebook · 321 stars · 256 forks · +5 stars in 24 hours · Trending score 1.17
-
4
sparklabx/sparklabx
Self-hosted multi-user Apache Spark notebooks — PySpark + Scala, per-user kernel pods, MinIO IAM storage, OAuth. One Helm install. Apache-2.0.
GitHub · TypeScript · 32 stars · 9 forks · +4 stars in 24 hours · Trending score 1.06
-
5
MrPowers/chispa
PySpark test helper methods with beautiful error messages
GitHub · Python · 778 stars · 86 forks · +2 stars in 24 hours · Trending score 1.03
-
6
lakehq/sail
Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.
GitHub · Rust · 3,506 stars · 246 forks · +1 stars in 24 hours · Trending score 1.02
-
7
ibis-project/ibis
the portable Python dataframe library
GitHub · Python · 6,706 stars · 791 forks · +1 stars in 24 hours · Trending score 0.99
-
8
Masoudjafaripour/Multimodal_Datasets_Generative_Reasoning
A repository for surveying, organizing, and prototyping dataset and benchmark construction pipelines for generative reasoning in multimodal large language models. It focuses on data-centric practices—creation, curation, and extension of multimodal datasets—with an emphasis on spatial, visual, and reasoning-oriented tasks.
GitHub · Jupyter Notebook · 15 stars · 0 forks · +3 stars in 24 hours · Trending score 0.92
-
9
NHSDigital/data-validation-engine
Data Validation Engine source code
GitHub · Python · 7 stars · 6 forks · +2 stars in 24 hours · Trending score 0.81
-
10
narwhals-dev/narwhals
Lightweight and extensible compatibility layer between dataframe libraries!
GitHub · Python · 1,756 stars · 281 forks · 0 stars in 24 hours · Trending score 0.74
-
11
apache/linkis
Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
GitHub · Java · 3,417 stars · 1,169 forks · 0 stars in 24 hours · Trending score 0.67
-
12
opendp/tumult-analytics
Tumult Analytics is a Python library for privately computing aggregate queries on tabular data. It is built atop the Tumult Core library.
GitHub · Python · 18 stars · 11 forks · +1 stars in 24 hours · Trending score 0.62
-
13
rafaelmatsuyama/FIAP-ABD-StreamingDataPipelineArchitectures
Enterprise Reference Architecture & interactive hands-on lab suite for Real-Time Stream Processing, Apache Spark Structured Streaming, Delta Lakehouse Medallion, and Serverless Apache Flink SQL.
GitHub · Jupyter Notebook · 6 stars · 26 forks · +1 stars in 24 hours · Trending score 0.61
-
14
ssanjaychandra123/data-engineering-patterns
600+ free patterns and concepts for data engineers on Azure Databricks, Microsoft Fabric, and PySpark. 12 books covering the full stack. Free forever.
GitHub · 93 stars · 39 forks · 0 stars in 24 hours · Trending score 0.58
-
15
shauryashaurya/The-Silmaril
Practice #ontology engineering with Python (and other languages).
GitHub · Jupyter Notebook · 39 stars · 9 forks · +1 stars in 24 hours · Trending score 0.45
-
16
sparkdq-community/sparkdq
A lightweight, declarative PySpark framework for data quality validation — check columns, rows, and entire datasets directly in your Spark pipelines
GitHub · Python · 77 stars · 10 forks · 0 stars in 24 hours · Trending score 0.29
-
17
DataFabric-Academy/ms-fabric-data-science
Hands-on Microsoft Fabric Data Science labs (FreshMart churn) with Thai study notes, PySpark/MLflow, and DP-604-aligned learning path.
GitHub · Jupyter Notebook · 99 stars · 3 forks · 0 stars in 24 hours · Trending score 0.14
-
18
animetanver-commits/fabric-unified-data-blueprint
Microsoft Fabric Unified Data Foundation with Databricks & Purview 2026
GitHub · HTML · 114 stars · 0 forks · 0 stars in 24 hours · Trending score 0.10
-
19
maurinruneacco/luau-diagrams-and-defects
Production Data Engineering Cloud Architecture Workflow Automation Portfolio 2026
GitHub · HTML · 69 stars · 0 forks · 0 stars in 24 hours · Trending score 0.10
-
20
opentargets/gentropy
Open Targets python framework for post-GWAS analysis
GitHub · Python · 62 stars · 17 forks · 0 stars in 24 hours · Trending score 0.09
-
21
mahmoudparsian/big-data-modeling
Big Data Modeling, MapReduce, Spark, PySpark @ Santa Clara University
GitHub · HTML · 167 stars · 144 forks · 0 stars in 24 hours · Trending score 0.06
-
22
khuyentran1401/codecut-blog
45+ production-ready tutorials on data science, MLOps, and AI tools. All code is executable and adaptable for real projects.
GitHub · Jupyter Notebook · 42 stars · 12 forks · 0 stars in 24 hours · Trending score 0.04
-
23
Matt17BR/openwrangler
Open Wrangler is an open-source visual dataframe editor for VS Code and editors based on it. Explore files or data loaded in memory, make changes through a GUI, and generate Python or R code.
GitHub · TypeScript · 7 stars · 1 forks · 0 stars in 24 hours · Trending score 0.04
-
24
apache/incubator-graphar
An open source, standard data file format for graph data storage and retrieval.
GitHub · Java · 383 stars · 98 forks · −1 stars in 24 hours · Trending score 0.03