pretraining repositories
Trending repositories tagged pretraining, ranked by trending score. Updated daily.
-
1
rasbt/LLMs-from-scratch
Implement a ChatGPT-like LLM in PyTorch from scratch, step by step
GitHub · Jupyter Notebook · 108,691 stars · 16,815 forks · +100 stars in 24 hours · Trending score 3.93
-
2
logitcai/tinybrain
A Chinese-first small language model built from scratch: open 60B-token dataset, tokenizer, mHC + Engram + MoE model, pretraining through RL, all in PyTorch. 从零做中文小模型,数据和代码全部开源
GitHub · Python · 252 stars · 9 forks · +232 stars in 24 hours · Trending score 2.95
-
3
OpenWAM-Official/OpenWAM
Official repository for "OpenWAM: An Open, Modular Exploration Towards Systematic World–Action Model Pretraining".
GitHub · Python · 1,210 stars · 89 forks · +6 stars in 24 hours · Trending score 1.51
-
4
JaydenTeoh/Soft-MTP
Soft-MTP: Multi-Token Prediction Done Right. KL gradient recovery for faster speculative decoding and better language model training.
GitHub · Python · 108 stars · 13 forks · +2 stars in 24 hours · Trending score 1.18
-
5
entanglr/awesome-vintage-llms
A curated list of vintage large language models — also called historical or time-capsule LLMs — trained from scratch on text from bounded historical periods, along with the papers, datasets, demos, and discussions surrounding them.
GitHub · 49 stars · 3 forks · 0 stars in 24 hours · Trending score 0.95
-
6
baidu-baige/LoongForge
A high-performance framework for training LLMs, VLMs, diffusion, and embodied models on NVIDIA GPUs and Kunlun XPUs. Supports Pi0.5, GR00T-N1.7, FastWAM, Cosmos3, DreamZero, DeepSeek-V4, GLM-5.3, Kimi-K3 and more.
GitHub · Python · 869 stars · 70 forks · 0 stars in 24 hours · Trending score 0.46
-
7
tum-pbs/tadpole
[NeurIPS2026] Flexible scientific foundation models trained with synthetic online data generation
GitHub · Python · 32 stars · 5 forks · 0 stars in 24 hours · Trending score 0.24
-
8
GaotangLi/Multi-Branch-Transformer
Multi-Branch Transformer: Scaling language models through parallel branches.
GitHub · Python · 9 stars · 0 forks · 0 stars in 24 hours · Trending score 0.09
-
9
lavishguptaa/llm-architecture-atlas
An open-source collection covering almost all major LLM architectures, built to explore, understand, and implement the building blocks of modern language models.
GitHub · Python · 11 stars · 1 forks · 0 stars in 24 hours · Trending score 0.07
-
10
AkashPriyadarshii/jev-curate
High-throughput synthetic and pretraining dataset sifter for TypeSafe Jev. Rust streaming core, Parquet and JSONL I/O, typed Choice/Score/Noul judgments, speculative fan-out, 24.0 rows/sec measured.
GitHub · Rust · 110 stars · 14 forks · 0 stars in 24 hours · Trending score 0.04
-
11
InternScience/GraphGen
GraphGen: Enhancing Supervised Fine-Tuning for LLMs with Knowledge-Driven Synthetic Data Generation
GitHub · Python · 1,231 stars · 100 forks · 0 stars in 24 hours · Trending score 0.02
-
12
sapientinc/HRM-Text
HRM-Text is a 1B text generation model based on the HRM architecture, strengthened by task completion and latent space reasoning.
GitHub · Python · 2,203 stars · 211 forks · 0 stars in 24 hours · Trending score 0.00
-
13
aryanap07/Timeless
Before the prompt, there was the architecture.
GitHub · Jupyter Notebook · 5 stars · 3 forks · 0 stars in 24 hours · Trending score 0.00
-
14
Enping-Hu/minimind-deep-dive
从 MiniMind 源码读起,再延伸到现代大模型技术体系的中文学习笔记。主线逐行精读预训练 / SFT / DPO / PPO / GRPO 与训练机制;附录 17 篇进阶卷覆盖量化、投机解码、RLHF 全景、模型代际史等 MiniMind 没涉及、但进阶绕不开的主题。
GitHub · Python · 216 stars · 13 forks · 0 stars in 24 hours · Trending score 0.00
-
15
GonzaloFuentes1/llm-learning-route
Roadmap and 44 hands-on guides (Spanish) for building LLMs: data, pretraining, post-training, evaluation and serving
GitHub · HTML · 11 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00
-
16
OpenWAM-Official/OpenWAM-Official.github.io
Project page for "OpenWAM: An Open, Modular Exploration Towards Systematic World–Action Model Pretraining".
GitHub · TypeScript · 8 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00
-
17
Physics-Scaling/GeoPT
Official Code for "GeoPT: Scaling Physics Simulation via Lifted Geometric Pre-Training" (ICML 2026) https://arxiv.org/pdf/2602.20399
GitHub · Python · 279 stars · 33 forks · 0 stars in 24 hours · Trending score 0.00
-
18
lucidrains/self-play-pretrain-zero-data
Implementation of Self-Play Pretraining with Zero Data
GitHub · Python · 10 stars · 1 forks · 0 stars in 24 hours · Trending score 0.00
-
19
MatthewK78/Rose
🌹 Rose: Range-Of-Slice Equilibration PyTorch optimizer. Stateless optimization through range-normalized gradient updates.
GitHub · Python · 78 stars · 5 forks · 0 stars in 24 hours · Trending score 0.00
-
20
yangqi0/petitgpt
A 124.6M LLM trained from scratch on 13B tokens on a single RTX 4090 — tokenizer, pretraining, post-training, evaluation, and reproducible inference.
GitHub · Python · 71 stars · 14 forks · 0 stars in 24 hours · Trending score 0.00
-
21
SobhanYasami/LLMs-from-scratch-by-me
Learning repo and future reference, following Build a Large Language Model (From Scratch) by Sebastian Raschka.
GitHub · HTML · 5 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00
-
22
pilot7747/bux
A moddable training framework for pre-training ablations.
GitHub · Python · 20 stars · 2 forks · 0 stars in 24 hours · Trending score 0.00
-
23
1023097618/train_llm
A small chat model trained from scratch: tokenizer → pretraining → SFT → RL → multimodal. 0.78B params, CORE 0.248, tool calling, vLLM. Weights on Hugging Face.
GitHub · Python · 20 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00