moe repositories

Trending repositories tagged moe, ranked by trending score. Updated daily.

  1. 1
    FareedKhan-dev/kimi-k3-in-c

    A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.

    GitHub · C · 12,485 stars · 1,932 forks · +181 stars in 24 hours · Trending score 4.37

  2. 2
    vllm-project/vllm

    A high-throughput and memory-efficient inference and serving engine for LLMs

    GitHub · Python · 97,255 stars · 25,073 forks · +109 stars in 24 hours · Trending score 4.26

  3. 3
    mw00/project-maya

    Run GLM-5.3 (744B), DeepSeek-V4.1-Flash (748B), MiniMax-M3.1-Flash (428B), GLM-5.3-Flash (321B) and Hy4-Flash (220B) on your own GPUs - from 8 GB of VRAM, NVIDIA and AMD, one dashboard, OpenAI- and Anthropic-compatible API.

    GitHub · C++ · 3,094 stars · 430 forks · +69 stars in 24 hours · Trending score 3.63

  4. 4
    wici-ai/inferssd

    Run models larger than your RAM or VRAM by streaming MoE experts from SSD. Built on llama.cpp.

    GitHub · C++ · 1,262 stars · 113 forks · +32 stars in 24 hours · Trending score 3.48

  5. 5
    FlashML-org/FreeToken

    FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.

    GitHub · Python · 17,331 stars · 1,721 forks · +44 stars in 24 hours · Trending score 3.43

  6. 6
    hiyouga/LlamaFactory

    Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

    GitHub · Python · 76,148 stars · 9,394 forks · +48 stars in 24 hours · Trending score 3.32

  7. 7
    Helldez/BigMoeOnEdge

    Run MoE models bigger than your RAM. Frontier-size MoE on a phone, laptop, edge. Lossless, on stock llama.cpp. An adaptive streaming optimizer. Any hardware, any OS, any model, any quantization, behind a local OpenAI-compatible server.

    GitHub · C++ · 1,033 stars · 107 forks · +23 stars in 24 hours · Trending score 3.07

  8. 8
    RustingStudio/RustingStrata

    Pure-Rust inference engine for Qwen GGUF models (Qwen3 to Qwen3.8, dense and MoE up to 770B+). CUDA and Vulkan (NVIDIA, AMD), hybrid CPU+GPU experts, MTP speculative decoding, OpenAI and Anthropic compatible server. Up to ~1.7x llama.cpp decode on a 12 GB RTX 3060.

    GitHub · Rust · 291 stars · 36 forks · +31 stars in 24 hours · Trending score 2.90

  9. 9
    flashinfer-ai/flashinfer

    FlashInfer: Kernel Library for LLM Serving

    GitHub · Cuda · 6,961 stars · 1,773 forks · +21 stars in 24 hours · Trending score 2.81

  10. 10
    NVIDIA/TensorRT-LLM

    TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.

    GitHub · Python · 15,105 stars · 2,944 forks · +21 stars in 24 hours · Trending score 2.65

  11. 11
    sangharshadhyeta/Strata-HotRod

    Run Qwen3.8-Flash-Next (125B) on an RTX 3060 12 GB with 30 GB of RAM, streaming experts from the SSD. 50-70 tok/s, 262K context. A fork of Strata.

    GitHub · C++ · 37 stars · 3 forks · +24 stars in 24 hours · Trending score 2.53

  12. 12
    modelscope/ms-swift

    Use PEFT or Full-parameter to CPT/SFT/DPO/GRPO 600+ LLMs (Qwen3.8, DeepSeek-V4, GLM-5.1, InternLM3, Llama4, ...) and 300+ MLLMs (Qwen3-VL, Qwen3-Omni, InternVL3.5, Ovis2.5, GLM5.3, Gemma4, Llava, Phi4, ...) (AAAI 2025).

    GitHub · Python · 16,135 stars · 1,817 forks · +12 stars in 24 hours · Trending score 2.42

  13. 13
    sergqwer/strata-nvfp4

    Strata fork: Qwen3.8-Flash-Next 125B MoE in NVFP4 on one RTX 20-50 card (12 GB+) and 64 GB of RAM or more. Setup installs our GPTQ quants (huihui abliterated, OrcaRouter) from Hugging Face. CPU experts on AVX-512 or AVX2 (+AVX-VNNI), all experts in RAM from 92 GB, a low-RAM mode below, vision, W4A8 prompts. Windows / CUDA 13.

    GitHub · C++ · 251 stars · 31 forks · +23 stars in 24 hours · Trending score 2.42

  14. 14
    prabanta-dev/janas

    37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.

    GitHub · C · 735 stars · 73 forks · +11 stars in 24 hours · Trending score 2.41

  15. 15
    Indras-Mirror/Xolotl

    Xolotl: run DeepSeek-V4.1-Flash, all 552B, as a local coding agent on one gaming PC (RTX 4090 + 110 GB RAM): ~33 tok/s on agent tasks, ~41 in fast mode, 512K context. Plus GLM-5.3-Flash, Qwen3.8 and LFM3-Omni. MoE experts tiered across VRAM, pinned RAM and NVMe, with Xolotl Direct NVMe->VRAM. OpenAI + Anthropic API.

    GitHub · C++ · 545 stars · 67 forks · +12 stars in 24 hours · Trending score 2.18

  16. 16
    sgl-project/sglang

    SGLang is a high-performance serving framework for large language models and multimodal models.

    GitHub · Python · 38,872 stars · 10,647 forks · −123 stars in 24 hours · Trending score 1.80

  17. 17
    davide-troiani/moeshift

    Memory efficient and high performance Mixture-of-Experts inference for Apple Silicon

    GitHub · C++ · 127 stars · 7 forks · +4 stars in 24 hours · Trending score 1.49

  18. 18
    NVIDIA/cudnn-frontend

    cuDNN Frontend is NVIDIA's modern, open-source entry point to the cuDNN library and a growing collection of high-performance open-source kernels.

    GitHub · Cuda · 1,007 stars · 340 forks · +3 stars in 24 hours · Trending score 1.47

  19. 19
    Bruce-Lee-LY/gc-moe

    GC MoE is a CuTe DSL implementation of FP8/FP4 mixture-of-experts inference with CUDA Green Contexts for NVIDIA Hopper and Blackwell GPUs.

    GitHub · Python · 36 stars · 5 forks · +4 stars in 24 hours · Trending score 1.45

  20. 20
    kacper-daftcode/vLLM-Moet

    Frontier MoE models on RTX PRO 6000 / RTX 5090 (sm_120): DeepSeek-V4.1-Flash on 4x RTX PRO 6000, as a GPU+CPU hybrid on 4x RTX 5090, and 2-bit experts on vLLM v0.24.0

    GitHub · Sass · 579 stars · 49 forks · +7 stars in 24 hours · Trending score 1.44

  21. 21
    Morrowmake/glm53-flash-cmp170hx-recipe

    GLM-5.3-Flash on 4× NVIDIA CMP 170HX with vLLM: up to 482.6 tok/s single-user, up to 943.7 tok/s at 8 users, 262K context, TP4 or PP4, one-command container install

    GitHub · Python · 93 stars · 14 forks · +1 stars in 24 hours · Trending score 1.42

  22. 22
    T-Crypt/halocline

    Sparse + Dense. Qwen4 family on one RTX 4090 (sm_89 only)

    GitHub · C++ · 18 stars · 1 forks · +12 stars in 24 hours · Trending score 1.35

  23. 23
    whitecircle/halo

    Halo is an open-source framework built by White Circle for training large language and multimodal models

    GitHub · Python · 868 stars · 50 forks · +2 stars in 24 hours · Trending score 1.32

  24. 24
    uccl-project/uccl

    UCCL is an efficient communication library for GPUs, covering collectives, P2P (e.g., KV cache transfer, RL weight transfer), and EP (e.g., GPU-driven)

    GitHub · C++ · 1,588 stars · 199 forks · +2 stars in 24 hours · Trending score 1.12