pyspark repositories

Trending repositories tagged pyspark, ranked by trending score. Updated daily.

  1. 1
    kengio/databricks-certification-study-guide

    Open-source community study guide for all six Databricks certifications (Data Engineer Associate / Professional, Data Analyst Associate, ML Associate / Professional, GenAI Engineer Associate). Aligned to the 2025-2026 official exam guides. Obsidian-flavoured Markdown; PRs welcome.

    GitHub · JavaScript · 78 stars · 36 forks · +26 stars in 24 hours · Trending score 2.58

  2. 2
    pavanbadempet/AI-Healthcare-System

    AI Healthcare System & Clinical Data Platform. Combines a Databricks Medallion Lakehouse with TabICLv2 tabular foundation models, calibrated tree ensembles (CatBoost, XGBoost), multi-agent RAG, and clinical decision support portal.

    GitHub · Python · 75 stars · 19 forks · +9 stars in 24 hours · Trending score 1.71

  3. 3
    zorost/AI-Engineering-Lab

    A free, self-paced 24-week AI engineering course: Python, machine learning, LLMs, RAG, fine-tuning, agents and MCP, Azure and Vertex and Bedrock, and Databricks. 43 runnable notebooks, one continuous case study. MIT licensed, no signup. By Zorost Intelligence AI Lab.

    GitHub · Jupyter Notebook · 321 stars · 256 forks · +5 stars in 24 hours · Trending score 1.17

  4. 4
    sparklabx/sparklabx

    Self-hosted multi-user Apache Spark notebooks — PySpark + Scala, per-user kernel pods, MinIO IAM storage, OAuth. One Helm install. Apache-2.0.

    GitHub · TypeScript · 32 stars · 9 forks · +4 stars in 24 hours · Trending score 1.06

  5. 5
    MrPowers/chispa

    PySpark test helper methods with beautiful error messages

    GitHub · Python · 778 stars · 86 forks · +2 stars in 24 hours · Trending score 1.03

  6. 6
    lakehq/sail

    Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

    GitHub · Rust · 3,506 stars · 246 forks · +1 stars in 24 hours · Trending score 1.02

  7. 7
    ibis-project/ibis

    the portable Python dataframe library

    GitHub · Python · 6,706 stars · 791 forks · +1 stars in 24 hours · Trending score 0.99

  8. 8
    Masoudjafaripour/Multimodal_Datasets_Generative_Reasoning

    A repository for surveying, organizing, and prototyping dataset and benchmark construction pipelines for generative reasoning in multimodal large language models. It focuses on data-centric practices—creation, curation, and extension of multimodal datasets—with an emphasis on spatial, visual, and reasoning-oriented tasks.

    GitHub · Jupyter Notebook · 15 stars · 0 forks · +3 stars in 24 hours · Trending score 0.92

  9. 9
    NHSDigital/data-validation-engine

    Data Validation Engine source code

    GitHub · Python · 7 stars · 6 forks · +2 stars in 24 hours · Trending score 0.81

  10. 10
    narwhals-dev/narwhals

    Lightweight and extensible compatibility layer between dataframe libraries!

    GitHub · Python · 1,756 stars · 281 forks · 0 stars in 24 hours · Trending score 0.74

  11. 11
    apache/linkis

    Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.

    GitHub · Java · 3,417 stars · 1,169 forks · 0 stars in 24 hours · Trending score 0.67

  12. 12
    opendp/tumult-analytics

    Tumult Analytics is a Python library for privately computing aggregate queries on tabular data. It is built atop the Tumult Core library.

    GitHub · Python · 18 stars · 11 forks · +1 stars in 24 hours · Trending score 0.62

  13. 13
    rafaelmatsuyama/FIAP-ABD-StreamingDataPipelineArchitectures

    Enterprise Reference Architecture & interactive hands-on lab suite for Real-Time Stream Processing, Apache Spark Structured Streaming, Delta Lakehouse Medallion, and Serverless Apache Flink SQL.

    GitHub · Jupyter Notebook · 6 stars · 26 forks · +1 stars in 24 hours · Trending score 0.61

  14. 14
    ssanjaychandra123/data-engineering-patterns

    600+ free patterns and concepts for data engineers on Azure Databricks, Microsoft Fabric, and PySpark. 12 books covering the full stack. Free forever.

    GitHub · 93 stars · 39 forks · 0 stars in 24 hours · Trending score 0.58

  15. 15
    shauryashaurya/The-Silmaril

    Practice #ontology engineering with Python (and other languages).

    GitHub · Jupyter Notebook · 39 stars · 9 forks · +1 stars in 24 hours · Trending score 0.45

  16. 16
    sparkdq-community/sparkdq

    A lightweight, declarative PySpark framework for data quality validation — check columns, rows, and entire datasets directly in your Spark pipelines

    GitHub · Python · 77 stars · 10 forks · 0 stars in 24 hours · Trending score 0.29

  17. 17
    DataFabric-Academy/ms-fabric-data-science

    Hands-on Microsoft Fabric Data Science labs (FreshMart churn) with Thai study notes, PySpark/MLflow, and DP-604-aligned learning path.

    GitHub · Jupyter Notebook · 99 stars · 3 forks · 0 stars in 24 hours · Trending score 0.14

  18. 18
    animetanver-commits/fabric-unified-data-blueprint

    Microsoft Fabric Unified Data Foundation with Databricks & Purview 2026

    GitHub · HTML · 114 stars · 0 forks · 0 stars in 24 hours · Trending score 0.10

  19. 19
    maurinruneacco/luau-diagrams-and-defects

    Production Data Engineering Cloud Architecture Workflow Automation Portfolio 2026

    GitHub · HTML · 69 stars · 0 forks · 0 stars in 24 hours · Trending score 0.10

  20. 20
    opentargets/gentropy

    Open Targets python framework for post-GWAS analysis

    GitHub · Python · 62 stars · 17 forks · 0 stars in 24 hours · Trending score 0.09

  21. 21
    mahmoudparsian/big-data-modeling

    Big Data Modeling, MapReduce, Spark, PySpark @ Santa Clara University

    GitHub · HTML · 167 stars · 144 forks · 0 stars in 24 hours · Trending score 0.06

  22. 22
    khuyentran1401/codecut-blog

    45+ production-ready tutorials on data science, MLOps, and AI tools. All code is executable and adaptable for real projects.

    GitHub · Jupyter Notebook · 42 stars · 12 forks · 0 stars in 24 hours · Trending score 0.04

  23. 23
    Matt17BR/openwrangler

    Open Wrangler is an open-source visual dataframe editor for VS Code and editors based on it. Explore files or data loaded in memory, make changes through a GUI, and generate Python or R code.

    GitHub · TypeScript · 7 stars · 1 forks · 0 stars in 24 hours · Trending score 0.04

  24. 24
    apache/incubator-graphar

    An open source, standard data file format for graph data storage and retrieval.

    GitHub · Java · 383 stars · 98 forks · −1 stars in 24 hours · Trending score 0.03