speculative-decoding repositories
Trending repositories tagged speculative-decoding, ranked by trending score. Updated daily.
-
1
incoai/splash
A local inference engine for Apple silicon, built around the model.
GitHub · C++ · 2,543 stars · 245 forks · +112 stars in 24 hours · Trending score 3.77
-
2
simplexai-labs/PolyStrata
Run large AI models on your own PC: Qwen3.8-Flash-Next on Strata's engine, and GLM 5.3 Flash, DeepSeek V4 Flash, LFM2.5, Qwen3.6, Ornith 1.5, Gemma 4, Tiel Coder and MiniCPM-V on a second engine written for PolyStrata. One-click install for Windows and Linux, OpenAI/Anthropic API on localhost.
GitHub · C++ · 1,280 stars · 166 forks · +60 stars in 24 hours · Trending score 3.63
-
3
synchopate/llama-modchip
Run 284B DeepSeek-V4-Flash (<59GiB RAM: 41.1 cold pp, 9.1 N=1 tg, up to 25.3 spec tok/s) & 30B MoEs (22.4 N=1 / 111.6 spec tok/s) on 64GB CPU (0 GPUs).
GitHub · Python · 1,387 stars · 134 forks · +38 stars in 24 hours · Trending score 3.55
-
4
JakeATX/llamAmpere
llama.cpp fork for significantly improved performance on Ampere generation Nvidia cards (especially RTX 3090 / 3090 Ti): example: 110+ tok/s over a 100K-token generation at temperature 1 for qwen3.8, supporting full 262K ctx. Supports low memory (12gb) configs for 200k+ ctx. 120+ TPS, 3300+ prefill for QFN 3.8.
GitHub · C++ · 403 stars · 50 forks · +87 stars in 24 hours · Trending score 3.35
-
5
RustingStudio/RustingStrata
Pure-Rust inference engine for Qwen GGUF models (Qwen3 to Qwen3.8, dense and MoE up to 770B+). CUDA and Vulkan (NVIDIA, AMD), hybrid CPU+GPU experts, MTP speculative decoding, OpenAI and Anthropic compatible server. Up to ~1.7x llama.cpp decode on a 12 GB RTX 3060.
GitHub · Rust · 291 stars · 36 forks · +31 stars in 24 hours · Trending score 2.90
-
6
sybil-solutions/inference-optimisation-skill
Agent skill for hill-climbing local LLM serving speed (decode, speculative decoding, prefill, KV, startup, TTFT) on any hardware and quantization
GitHub · Python · 1,284 stars · 114 forks · +372 stars in 24 hours · Trending score 2.77
-
7
peonist-ai/halogen-flash-server
Local inference that behaves like a frontier API: Qwen3.8-Flash-Next on AMD Strix Halo plus small NPU models, over the OpenAI and Anthropic APIs on one port.
GitHub · Shell · 1,403 stars · 101 forks · +17 stars in 24 hours · Trending score 2.66
-
8
nace-ai/kapitsa
Blazing fast Apple Silicon inference engine optimized for Qwen & Nemotron. OpenAI + Anthropic API compatible. Built by folks from nace.ai
GitHub · C++ · 833 stars · 115 forks · +14 stars in 24 hours · Trending score 2.56
-
9
prabanta-dev/janas
37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.
GitHub · C · 735 stars · 73 forks · +11 stars in 24 hours · Trending score 2.41
-
10
vllm-project/speculators
A unified library for building, evaluating, and storing speculative decoding algorithms for LLM inference in vLLM
GitHub · Python · 967 stars · 298 forks · +7 stars in 24 hours · Trending score 2.09
-
11
raullenchai/Rapid-MLX
Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm) on the same weights.
GitHub · Python · 4,166 stars · 446 forks · +7 stars in 24 hours · Trending score 2.04
-
12
ahtx/kindi
Kindi 0.6.1: fast local LLM inference on low-cost GPUs and CPUs. DeepSeek-V4.1-Flash (552B) at 18.7 tok/s on one RTX 3060 12 GB with a dual-Xeon DDR4 server. Ternary Bonsai 2B-27B, AH-Nanbeige4.3-33B, Nemotron 3.5 Lightning 30B-A3B. NVIDIA from the Quadro K420 to the RTX 50 series, AMD Radeon graphics, Windows and Linux. OpenAI-compatible server.
GitHub · Python · 195 stars · 41 forks · +6 stars in 24 hours · Trending score 1.95
-
13
bpx03/QEngine
Fast open-source (MIT) LLM inference engine in C++/CUDA for RTX 3090-class GPUs: Qwen3.8-Flash-Next (~1,000 tok/s with 8 agents on 3x RTX 3090) and Qwen3.8-27B, OpenAI/Anthropic-compatible API.
GitHub · Cuda · 75 stars · 8 forks · +4 stars in 24 hours · Trending score 1.75
-
14
TandemLLM/TandemEngine
TandemEngine: local inference that drafts and verifies its own tokens, behind an OpenAI-compatible API.
GitHub · Python · 82 stars · 17 forks · +8 stars in 24 hours · Trending score 1.73
-
15
youssofal/MTPLX
The fastest way to run Qwen 3.8 Flash Next, Qwen 3.8 27B and Ternary Bonsai 2 27B on a Mac: 125 tok/s in OpenCode on an M5 Max, and a 27B model on 16 GB Macs. Native MTP speculative decoding on Apple Silicon, exact at any temperature. OpenAI and Anthropic compatible local server.
GitHub · Python · 2,726 stars · 214 forks · +6 stars in 24 hours · Trending score 1.72
-
16
Luce-Org/lucebox
Lucebox inference engine for heterogeneous hardware & consumer GPUs
GitHub · C++ · 3,018 stars · 299 forks · +3 stars in 24 hours · Trending score 1.63
-
17
hasso5703/alzeo
Alzeo is an open-source installer that turns an NVIDIA DGX Spark, or any GB10 machine (ASUS, Dell, HP, Lenovo, MSI, GIGABYTE, Acer), into a private AI server in one command: measured model lanes, OpenAI and Anthropic APIs, and a local cockpit.
GitHub · Python · 270 stars · 24 forks · +6 stars in 24 hours · Trending score 1.60
-
18
Tencent/AngelSlim
Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.
GitHub · Python · 1,806 stars · 199 forks · +4 stars in 24 hours · Trending score 1.51
-
19
hyukudan/strix-halo-engine-lab
Exact LLM inference engine written from scratch for AMD Strix Halo (gfx1151, ROCm/HIP). Qwen3.8-Flash-Next on one node or two replicas; DeepSeek V4.1 Flash across two nodes over USB4 with an optional RTX PRO. Speculative decoding bit-identical to greedy, up to 255 tok/s on coding chat. MiniMax and GLM next.
GitHub · C++ · 37 stars · 6 forks · +4 stars in 24 hours · Trending score 1.48
-
20
jaedync/crucible-strix-halo
Run Qwen3.8-Flash-Next on one AMD Strix Halo (Ryzen AI Max+ 395, gfx1151). EXL3 4.05 and 3.05 bpw packs, own ROCm/HIP kernels, OpenAI-compatible server, 4 sessions of 262K context, setup guide included.
GitHub · Python · 35 stars · 7 forks · +2 stars in 24 hours · Trending score 1.45
-
21
Morrowmake/glm53-flash-cmp170hx-recipe
GLM-5.3-Flash on 4× NVIDIA CMP 170HX with vLLM: up to 482.6 tok/s single-user, up to 943.7 tok/s at 8 users, 262K context, TP4 or PP4, one-command container install
GitHub · Python · 93 stars · 14 forks · +1 stars in 24 hours · Trending score 1.42
-
22
ARahim3/kaggle-tpu-lab
Frontier-class open models on a free Kaggle TPU v5e-8: GLM-5.3-Flash 320B MoE (~165 tok/s, our own JAX engine) and Qwen3.8-27B bf16 (~130 tok/s), 262k context, prefix caching. Works with Claude Code, Codex, opencode and pi.
GitHub · Python · 813 stars · 110 forks · +2 stars in 24 hours · Trending score 1.32
-
23
JaydenTeoh/Soft-MTP
Soft-MTP: Multi-Token Prediction Done Right. KL gradient recovery for faster speculative decoding and better language model training.
GitHub · Python · 108 stars · 13 forks · +2 stars in 24 hours · Trending score 1.18
-
24
tuan3w/qwen-warp
Fast Qwen3.8-27B for one RTX 3090. 200k context, 100+ tok/s. Custom CUDA decode engine, OpenAI-compatible API.
GitHub · Python · 49 stars · 6 forks · +3 stars in 24 hours · Trending score 1.16