speculative-decoding repositories

Trending repositories tagged speculative-decoding, ranked by trending score. Updated daily.

  1. 1
    incoai/splash

    A local inference engine for Apple silicon, built around the model.

    GitHub · C++ · 2,543 stars · 245 forks · +112 stars in 24 hours · Trending score 3.77

  2. 2
    simplexai-labs/PolyStrata

    Run large AI models on your own PC: Qwen3.8-Flash-Next on Strata's engine, and GLM 5.3 Flash, DeepSeek V4 Flash, LFM2.5, Qwen3.6, Ornith 1.5, Gemma 4, Tiel Coder and MiniCPM-V on a second engine written for PolyStrata. One-click install for Windows and Linux, OpenAI/Anthropic API on localhost.

    GitHub · C++ · 1,280 stars · 166 forks · +60 stars in 24 hours · Trending score 3.63

  3. 3
    synchopate/llama-modchip

    Run 284B DeepSeek-V4-Flash (<59GiB RAM: 41.1 cold pp, 9.1 N=1 tg, up to 25.3 spec tok/s) & 30B MoEs (22.4 N=1 / 111.6 spec tok/s) on 64GB CPU (0 GPUs).

    GitHub · Python · 1,387 stars · 134 forks · +38 stars in 24 hours · Trending score 3.55

  4. 4
    JakeATX/llamAmpere

    llama.cpp fork for significantly improved performance on Ampere generation Nvidia cards (especially RTX 3090 / 3090 Ti): example: 110+ tok/s over a 100K-token generation at temperature 1 for qwen3.8, supporting full 262K ctx. Supports low memory (12gb) configs for 200k+ ctx. 120+ TPS, 3300+ prefill for QFN 3.8.

    GitHub · C++ · 403 stars · 50 forks · +87 stars in 24 hours · Trending score 3.35

  5. 5
    RustingStudio/RustingStrata

    Pure-Rust inference engine for Qwen GGUF models (Qwen3 to Qwen3.8, dense and MoE up to 770B+). CUDA and Vulkan (NVIDIA, AMD), hybrid CPU+GPU experts, MTP speculative decoding, OpenAI and Anthropic compatible server. Up to ~1.7x llama.cpp decode on a 12 GB RTX 3060.

    GitHub · Rust · 291 stars · 36 forks · +31 stars in 24 hours · Trending score 2.90

  6. 6
    sybil-solutions/inference-optimisation-skill

    Agent skill for hill-climbing local LLM serving speed (decode, speculative decoding, prefill, KV, startup, TTFT) on any hardware and quantization

    GitHub · Python · 1,284 stars · 114 forks · +372 stars in 24 hours · Trending score 2.77

  7. 7
    peonist-ai/halogen-flash-server

    Local inference that behaves like a frontier API: Qwen3.8-Flash-Next on AMD Strix Halo plus small NPU models, over the OpenAI and Anthropic APIs on one port.

    GitHub · Shell · 1,403 stars · 101 forks · +17 stars in 24 hours · Trending score 2.66

  8. 8
    nace-ai/kapitsa

    Blazing fast Apple Silicon inference engine optimized for Qwen & Nemotron. OpenAI + Anthropic API compatible. Built by folks from nace.ai

    GitHub · C++ · 833 stars · 115 forks · +14 stars in 24 hours · Trending score 2.56

  9. 9
    prabanta-dev/janas

    37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.

    GitHub · C · 735 stars · 73 forks · +11 stars in 24 hours · Trending score 2.41

  10. 10
    vllm-project/speculators

    A unified library for building, evaluating, and storing speculative decoding algorithms for LLM inference in vLLM

    GitHub · Python · 967 stars · 298 forks · +7 stars in 24 hours · Trending score 2.09

  11. 11
    raullenchai/Rapid-MLX

    Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm) on the same weights.

    GitHub · Python · 4,166 stars · 446 forks · +7 stars in 24 hours · Trending score 2.04

  12. 12
    ahtx/kindi

    Kindi 0.6.1: fast local LLM inference on low-cost GPUs and CPUs. DeepSeek-V4.1-Flash (552B) at 18.7 tok/s on one RTX 3060 12 GB with a dual-Xeon DDR4 server. Ternary Bonsai 2B-27B, AH-Nanbeige4.3-33B, Nemotron 3.5 Lightning 30B-A3B. NVIDIA from the Quadro K420 to the RTX 50 series, AMD Radeon graphics, Windows and Linux. OpenAI-compatible server.

    GitHub · Python · 195 stars · 41 forks · +6 stars in 24 hours · Trending score 1.95

  13. 13
    bpx03/QEngine

    Fast open-source (MIT) LLM inference engine in C++/CUDA for RTX 3090-class GPUs: Qwen3.8-Flash-Next (~1,000 tok/s with 8 agents on 3x RTX 3090) and Qwen3.8-27B, OpenAI/Anthropic-compatible API.

    GitHub · Cuda · 75 stars · 8 forks · +4 stars in 24 hours · Trending score 1.75

  14. 14
    TandemLLM/TandemEngine

    TandemEngine: local inference that drafts and verifies its own tokens, behind an OpenAI-compatible API.

    GitHub · Python · 82 stars · 17 forks · +8 stars in 24 hours · Trending score 1.73

  15. 15
    youssofal/MTPLX

    The fastest way to run Qwen 3.8 Flash Next, Qwen 3.8 27B and Ternary Bonsai 2 27B on a Mac: 125 tok/s in OpenCode on an M5 Max, and a 27B model on 16 GB Macs. Native MTP speculative decoding on Apple Silicon, exact at any temperature. OpenAI and Anthropic compatible local server.

    GitHub · Python · 2,726 stars · 214 forks · +6 stars in 24 hours · Trending score 1.72

  16. 16
    Luce-Org/lucebox

    Lucebox inference engine for heterogeneous hardware & consumer GPUs

    GitHub · C++ · 3,018 stars · 299 forks · +3 stars in 24 hours · Trending score 1.63

  17. 17
    hasso5703/alzeo

    Alzeo is an open-source installer that turns an NVIDIA DGX Spark, or any GB10 machine (ASUS, Dell, HP, Lenovo, MSI, GIGABYTE, Acer), into a private AI server in one command: measured model lanes, OpenAI and Anthropic APIs, and a local cockpit.

    GitHub · Python · 270 stars · 24 forks · +6 stars in 24 hours · Trending score 1.60

  18. 18
    Tencent/AngelSlim

    Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.

    GitHub · Python · 1,806 stars · 199 forks · +4 stars in 24 hours · Trending score 1.51

  19. 19
    hyukudan/strix-halo-engine-lab

    Exact LLM inference engine written from scratch for AMD Strix Halo (gfx1151, ROCm/HIP). Qwen3.8-Flash-Next on one node or two replicas; DeepSeek V4.1 Flash across two nodes over USB4 with an optional RTX PRO. Speculative decoding bit-identical to greedy, up to 255 tok/s on coding chat. MiniMax and GLM next.

    GitHub · C++ · 37 stars · 6 forks · +4 stars in 24 hours · Trending score 1.48

  20. 20
    jaedync/crucible-strix-halo

    Run Qwen3.8-Flash-Next on one AMD Strix Halo (Ryzen AI Max+ 395, gfx1151). EXL3 4.05 and 3.05 bpw packs, own ROCm/HIP kernels, OpenAI-compatible server, 4 sessions of 262K context, setup guide included.

    GitHub · Python · 35 stars · 7 forks · +2 stars in 24 hours · Trending score 1.45

  21. 21
    Morrowmake/glm53-flash-cmp170hx-recipe

    GLM-5.3-Flash on 4× NVIDIA CMP 170HX with vLLM: up to 482.6 tok/s single-user, up to 943.7 tok/s at 8 users, 262K context, TP4 or PP4, one-command container install

    GitHub · Python · 93 stars · 14 forks · +1 stars in 24 hours · Trending score 1.42

  22. 22
    ARahim3/kaggle-tpu-lab

    Frontier-class open models on a free Kaggle TPU v5e-8: GLM-5.3-Flash 320B MoE (~165 tok/s, our own JAX engine) and Qwen3.8-27B bf16 (~130 tok/s), 262k context, prefix caching. Works with Claude Code, Codex, opencode and pi.

    GitHub · Python · 813 stars · 110 forks · +2 stars in 24 hours · Trending score 1.32

  23. 23
    JaydenTeoh/Soft-MTP

    Soft-MTP: Multi-Token Prediction Done Right. KL gradient recovery for faster speculative decoding and better language model training.

    GitHub · Python · 108 stars · 13 forks · +2 stars in 24 hours · Trending score 1.18

  24. 24
    tuan3w/qwen-warp

    Fast Qwen3.8-27B for one RTX 3090. 200k context, 100+ tok/s. Custom CUDA decode engine, OpenAI-compatible API.

    GitHub · Python · 49 stars · 6 forks · +3 stars in 24 hours · Trending score 1.16