quantization repositories

Trending repositories tagged quantization, ranked by trending score. Updated daily.

  1. 1
    FareedKhan-dev/kimi-k3-in-c

    A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.

    GitHub · C · 12,485 stars · 1,932 forks · +181 stars in 24 hours · Trending score 4.37

  2. 2
    hyperionaidev/smolmaxxing

    maxxing the ability to run different LLMs on the smallest devices possible

    GitHub · C · 1,035 stars · 54 forks · +65 stars in 24 hours · Trending score 3.88

  3. 3
    mw00/project-maya

    Run GLM-5.3 (744B), DeepSeek-V4.1-Flash (748B), MiniMax-M3.1-Flash (428B), GLM-5.3-Flash (321B) and Hy4-Flash (220B) on your own GPUs - from 8 GB of VRAM, NVIDIA and AMD, one dashboard, OpenAI- and Anthropic-compatible API.

    GitHub · C++ · 3,094 stars · 430 forks · +69 stars in 24 hours · Trending score 3.63

  4. 4
    synchopate/llama-modchip

    Run 284B DeepSeek-V4-Flash (<59GiB RAM: 41.1 cold pp, 9.1 N=1 tg, up to 25.3 spec tok/s) & 30B MoEs (22.4 N=1 / 111.6 spec tok/s) on 64GB CPU (0 GPUs).

    GitHub · Python · 1,387 stars · 134 forks · +38 stars in 24 hours · Trending score 3.55

  5. 5
    JakeATX/llamAmpere

    llama.cpp fork for significantly improved performance on Ampere generation Nvidia cards (especially RTX 3090 / 3090 Ti): example: 110+ tok/s over a 100K-token generation at temperature 1 for qwen3.8, supporting full 262K ctx. Supports low memory (12gb) configs for 200k+ ctx. 120+ TPS, 3300+ prefill for QFN 3.8.

    GitHub · C++ · 403 stars · 50 forks · +87 stars in 24 hours · Trending score 3.35

  6. 6
    hiyouga/LlamaFactory

    Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

    GitHub · Python · 76,148 stars · 9,394 forks · +48 stars in 24 hours · Trending score 3.32

  7. 7
    huawei-csl/MACaQ

    SOTA model-agnostic quantization for GGUF, local inference and beyond. Quantize and run text, image, video and audio models with llama.cpp, vLLM, Diffusers and PyTorch.

    GitHub · C++ · 380 stars · 22 forks · +22 stars in 24 hours · Trending score 2.98

  8. 8
    animede/Diffusers_minimax-h3

    MiniMax-H3 (video + synchronized audio, 33B) on diffusers — 9.7s per still, 44.2s per 5s clip on 48GB, with every speedup verified byte-identical

    GitHub · Python · 61 stars · 6 forks · +25 stars in 24 hours · Trending score 2.61

  9. 9
    prabanta-dev/janas

    37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.

    GitHub · C · 735 stars · 73 forks · +11 stars in 24 hours · Trending score 2.41

  10. 10
    Indras-Mirror/Xolotl

    Xolotl: run DeepSeek-V4.1-Flash, all 552B, as a local coding agent on one gaming PC (RTX 4090 + 110 GB RAM): ~33 tok/s on agent tasks, ~41 in fast mode, 512K context. Plus GLM-5.3-Flash, Qwen3.8 and LFM3-Omni. MoE experts tiered across VRAM, pinned RAM and NVMe, with Xolotl Direct NVMe->VRAM. OpenAI + Anthropic API.

    GitHub · C++ · 545 stars · 67 forks · +12 stars in 24 hours · Trending score 2.18

  11. 11
    Anionex/h3-image-mlx

    Faster local MiniMax H3 Image generation and editing on Apple Silicon with MLX, with ultra-low ram

    GitHub · Python · 80 stars · 11 forks · +6 stars in 24 hours · Trending score 2.17

  12. 12
    hiruhiru2005/sync-strata

    A one-click local LLM app for Windows and Linux: Qwen3.8-Flash-Next with the Sync quantizations (XS, S, M), based on Strata.

    GitHub · C++ · 69 stars · 5 forks · +48 stars in 24 hours · Trending score 2.14

  13. 13
    amitshekhariitbhu/ai-engineering-interview-questions

    Your Cheat Sheet for AI Engineering Interview – Questions and Answers.

    GitHub · Markdown · 3,679 stars · 656 forks · +7 stars in 24 hours · Trending score 2.10

  14. 14
    vllm-project/llm-compressor

    State-of-the-art LLM compression, built for production inference with vLLM

    GitHub · Python · 3,977 stars · 739 forks · +7 stars in 24 hours · Trending score 1.86

  15. 15
    TandemLLM/TandemEngine

    TandemEngine: local inference that drafts and verifies its own tokens, behind an OpenAI-compatible API.

    GitHub · Python · 82 stars · 17 forks · +8 stars in 24 hours · Trending score 1.73

  16. 16
    warpfront/hipfire

    RDNA-native LLM inference engine in Rust.

    GitHub · Rust · 819 stars · 109 forks · +5 stars in 24 hours · Trending score 1.64

  17. 17
    AI-Hypercomputer/tpu-codesign

    Hardware-grounded model architecture design principles, MoE co-design patterns, low-precision numerics, and parallelism guides for training and serving on Google Cloud TPUs.

    GitHub · 90 stars · 7 forks · +25 stars in 24 hours · Trending score 1.58

  18. 18
    RyanCodrai/turbovec

    A vector index built on TurboQuant, written in Rust with Python bindings

    GitHub · Rust · 17,503 stars · 1,483 forks · +6 stars in 24 hours · Trending score 1.51

  19. 19
    Tencent/AngelSlim

    Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.

    GitHub · Python · 1,806 stars · 199 forks · +4 stars in 24 hours · Trending score 1.51

  20. 20
    jaedync/crucible-strix-halo

    Run Qwen3.8-Flash-Next on one AMD Strix Halo (Ryzen AI Max+ 395, gfx1151). EXL3 4.05 and 3.05 bpw packs, own ROCm/HIP kernels, OpenAI-compatible server, 4 sessions of 262K context, setup guide included.

    GitHub · Python · 35 stars · 7 forks · +2 stars in 24 hours · Trending score 1.45

  21. 21
    qdrant/skills

    Agent skills for Qdrant vector search: scaling, performance optimization, search quality, monitoring, deployment, model migration, version upgrades, and SDK usage across Python, TypeScript, Rust, Go, .NET, Java

    GitHub · Python · 271 stars · 31 forks · +7 stars in 24 hours · Trending score 1.44

  22. 22
    IST-DASLab/GSQ

    Gumbel-Softmax post-training quantization for LLMs (1–3 bit scalar, INT/GGUF-compatible).

    GitHub · Python · 143 stars · 21 forks · 0 stars in 24 hours · Trending score 1.38

  23. 23
    datawhalechina/llm-algo-leetcode

    Hands-on LLM algorithms and systems with PyTorch, Triton & CUDA.《大模型算法与系统教程》:模型架构与训练、后训练、推理加速、性能与算子优化,配套 Notebook 练习、答案解析、测试与实验。

    GitHub · Jupyter Notebook · 797 stars · 162 forks · 0 stars in 24 hours · Trending score 1.38

  24. 24
    whitecircle/halo

    Halo is an open-source framework built by White Circle for training large language and multimodal models

    GitHub · Python · 868 stars · 50 forks · +2 stars in 24 hours · Trending score 1.32