pretraining repositories

Trending repositories tagged pretraining, ranked by trending score. Updated daily.

  1. 1
    rasbt/LLMs-from-scratch

    Implement a ChatGPT-like LLM in PyTorch from scratch, step by step

    GitHub · Jupyter Notebook · 108,691 stars · 16,815 forks · +100 stars in 24 hours · Trending score 3.93

  2. 2
    logitcai/tinybrain

    A Chinese-first small language model built from scratch: open 60B-token dataset, tokenizer, mHC + Engram + MoE model, pretraining through RL, all in PyTorch. 从零做中文小模型,数据和代码全部开源

    GitHub · Python · 252 stars · 9 forks · +232 stars in 24 hours · Trending score 2.95

  3. 3
    OpenWAM-Official/OpenWAM

    Official repository for "OpenWAM: An Open, Modular Exploration Towards Systematic World–Action Model Pretraining".

    GitHub · Python · 1,210 stars · 89 forks · +6 stars in 24 hours · Trending score 1.51

  4. 4
    JaydenTeoh/Soft-MTP

    Soft-MTP: Multi-Token Prediction Done Right. KL gradient recovery for faster speculative decoding and better language model training.

    GitHub · Python · 108 stars · 13 forks · +2 stars in 24 hours · Trending score 1.18

  5. 5
    entanglr/awesome-vintage-llms

    A curated list of vintage large language models — also called historical or time-capsule LLMs — trained from scratch on text from bounded historical periods, along with the papers, datasets, demos, and discussions surrounding them.

    GitHub · 49 stars · 3 forks · 0 stars in 24 hours · Trending score 0.95

  6. 6
    baidu-baige/LoongForge

    A high-performance framework for training LLMs, VLMs, diffusion, and embodied models on NVIDIA GPUs and Kunlun XPUs. Supports Pi0.5, GR00T-N1.7, FastWAM, Cosmos3, DreamZero, DeepSeek-V4, GLM-5.3, Kimi-K3 and more.

    GitHub · Python · 869 stars · 70 forks · 0 stars in 24 hours · Trending score 0.46

  7. 7
    tum-pbs/tadpole

    [NeurIPS2026] Flexible scientific foundation models trained with synthetic online data generation

    GitHub · Python · 32 stars · 5 forks · 0 stars in 24 hours · Trending score 0.24

  8. 8
    GaotangLi/Multi-Branch-Transformer

    Multi-Branch Transformer: Scaling language models through parallel branches.

    GitHub · Python · 9 stars · 0 forks · 0 stars in 24 hours · Trending score 0.09

  9. 9
    lavishguptaa/llm-architecture-atlas

    An open-source collection covering almost all major LLM architectures, built to explore, understand, and implement the building blocks of modern language models.

    GitHub · Python · 11 stars · 1 forks · 0 stars in 24 hours · Trending score 0.07

  10. 10
    AkashPriyadarshii/jev-curate

    High-throughput synthetic and pretraining dataset sifter for TypeSafe Jev. Rust streaming core, Parquet and JSONL I/O, typed Choice/Score/Noul judgments, speculative fan-out, 24.0 rows/sec measured.

    GitHub · Rust · 110 stars · 14 forks · 0 stars in 24 hours · Trending score 0.04

  11. 11
    InternScience/GraphGen

    GraphGen: Enhancing Supervised Fine-Tuning for LLMs with Knowledge-Driven Synthetic Data Generation

    GitHub · Python · 1,231 stars · 100 forks · 0 stars in 24 hours · Trending score 0.02

  12. 12
    sapientinc/HRM-Text

    HRM-Text is a 1B text generation model based on the HRM architecture, strengthened by task completion and latent space reasoning.

    GitHub · Python · 2,203 stars · 211 forks · 0 stars in 24 hours · Trending score 0.00

  13. 13
    aryanap07/Timeless

    Before the prompt, there was the architecture.

    GitHub · Jupyter Notebook · 5 stars · 3 forks · 0 stars in 24 hours · Trending score 0.00

  14. 14
    Enping-Hu/minimind-deep-dive

    从 MiniMind 源码读起,再延伸到现代大模型技术体系的中文学习笔记。主线逐行精读预训练 / SFT / DPO / PPO / GRPO 与训练机制;附录 17 篇进阶卷覆盖量化、投机解码、RLHF 全景、模型代际史等 MiniMind 没涉及、但进阶绕不开的主题。

    GitHub · Python · 216 stars · 13 forks · 0 stars in 24 hours · Trending score 0.00

  15. 15
    GonzaloFuentes1/llm-learning-route

    Roadmap and 44 hands-on guides (Spanish) for building LLMs: data, pretraining, post-training, evaluation and serving

    GitHub · HTML · 11 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00

  16. 16
    OpenWAM-Official/OpenWAM-Official.github.io

    Project page for "OpenWAM: An Open, Modular Exploration Towards Systematic World–Action Model Pretraining".

    GitHub · TypeScript · 8 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00

  17. 17
    Physics-Scaling/GeoPT

    Official Code for "GeoPT: Scaling Physics Simulation via Lifted Geometric Pre-Training" (ICML 2026) https://arxiv.org/pdf/2602.20399

    GitHub · Python · 279 stars · 33 forks · 0 stars in 24 hours · Trending score 0.00

  18. 18
    lucidrains/self-play-pretrain-zero-data

    Implementation of Self-Play Pretraining with Zero Data

    GitHub · Python · 10 stars · 1 forks · 0 stars in 24 hours · Trending score 0.00

  19. 19
    MatthewK78/Rose

    🌹 Rose: Range-Of-Slice Equilibration PyTorch optimizer. Stateless optimization through range-normalized gradient updates.

    GitHub · Python · 78 stars · 5 forks · 0 stars in 24 hours · Trending score 0.00

  20. 20
    yangqi0/petitgpt

    A 124.6M LLM trained from scratch on 13B tokens on a single RTX 4090 — tokenizer, pretraining, post-training, evaluation, and reproducible inference.

    GitHub · Python · 71 stars · 14 forks · 0 stars in 24 hours · Trending score 0.00

  21. 21
    SobhanYasami/LLMs-from-scratch-by-me

    Learning repo and future reference, following Build a Large Language Model (From Scratch) by Sebastian Raschka.

    GitHub · HTML · 5 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00

  22. 22
    pilot7747/bux

    A moddable training framework for pre-training ablations.

    GitHub · Python · 20 stars · 2 forks · 0 stars in 24 hours · Trending score 0.00

  23. 23
    1023097618/train_llm

    A small chat model trained from scratch: tokenizer → pretraining → SFT → RL → multimodal. 0.78B params, CORE 0.248, tool calling, vLLM. Weights on Hugging Face.

    GitHub · Python · 20 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00