inference-optimization repositories

Trending repositories tagged inference-optimization, ranked by trending score. Updated daily.

  1. 1
    qr0n/p100-llama-cpp

    Patches for running llama.cpp fast on Tesla P100 (GP100/sm_60)

    GitHub · C++ · 37 stars · 4 forks · +22 stars in 24 hours · Trending score 2.25

  2. 2
    modular/llm-inference-handbook

    Everything you need to know about LLM inference

    GitHub · TypeScript · 489 stars · 61 forks · +8 stars in 24 hours · Trending score 1.62

  3. 3
    AI-Hypercomputer/tpu-codesign

    Hardware-grounded model architecture design principles, MoE co-design patterns, low-precision numerics, and parallelism guides for training and serving on Google Cloud TPUs.

    GitHub · 93 stars · 7 forks · +19 stars in 24 hours · Trending score 1.44

  4. 4
    google/XNNPACK

    High-efficiency floating-point neural network inference operators for mobile, server, and Web

    GitHub · C · 2,507 stars · 595 forks · +2 stars in 24 hours · Trending score 1.42

  5. 5
    datawhalechina/llm-algo-leetcode

    Hands-on LLM algorithms and systems with PyTorch, Triton & CUDA.《大模型算法与系统教程》:模型架构与训练、后训练、推理加速、性能与算子优化,配套 Notebook 练习、答案解析、测试与实验。

    GitHub · Jupyter Notebook · 797 stars · 162 forks · 0 stars in 24 hours · Trending score 1.37

  6. 6
    Hiruynk/AnifLive-TTS

    A low-latency, high-quality multilingual voice-cloning TTS runtime

    GitHub · Python · 56 stars · 4 forks · 0 stars in 24 hours · Trending score 1.08

  7. 7
    Superone77/FlashLoop

    FlashLoop: Fast and Memory-Efficient Looped Transformers via Lazy Updates

    GitHub · Python · 30 stars · 4 forks · 0 stars in 24 hours · Trending score 0.78

  8. 8
    Lowaltitudeflight/local-llm-tuning-guide

    A decision library for running local LLMs on multi-GPU machines without NVLink: parallelism, communication, quantization, speculative decoding, KV cache. Every conclusion carries its evidence grade and reproducible conditions.

    GitHub · Python · 105 stars · 0 forks · 0 stars in 24 hours · Trending score 0.63

  9. 9
    zhangshuibai/molten

    Run a coding model at 300+ tok/s on your Mac. Native Rust + Metal inference engine for LiquidAI LFM2.5-24B-A2B (MLX 4-bit) on Apple Silicon.

    GitHub · Metal · 29 stars · 2 forks · +1 stars in 24 hours · Trending score 0.53

  10. 10
    syv-ai/HyperQwen

    Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.

    GitHub · Python · 2,696 stars · 340 forks · 0 stars in 24 hours · Trending score 0.48

  11. 11
    Brontir/NInferno

    High-performance single-GPU inference for (Qwen, Gemma, Nemotron & other) checkpoints and GPUs.

    GitHub · C++ · 8 stars · 0 forks · 0 stars in 24 hours · Trending score 0.32

  12. 12
    david-xinyuwei/david-share

    AI Agents & Deep Learning: code, benchmarks and explainers. Building and operating agents on Microsoft Foundry and Azure; fine-tuning, inference optimization, precision and GPU benchmarks on NVIDIA and AMD; multimodal models.

    GitHub · Jupyter Notebook · 451 stars · 92 forks · 0 stars in 24 hours · Trending score 0.19

  13. 13
    B-A-M-N/SOLLOL

    Super Ollama Load Balancer - Performance-aware routing for distributed Ollama deployments with Ray, Dask, and adaptive metrics

    GitHub · Python · 6 stars · 5 forks · 0 stars in 24 hours · Trending score 0.15

  14. 14
    redhat-performance/configiq

    Distributed inference configuration tools

    GitHub · TypeScript · 23 stars · 13 forks · 0 stars in 24 hours · Trending score 0.09

  15. 15
    truespar/paddock

    Native Rust inference server for open models on NVIDIA and Apple Silicon. OpenAI- and Anthropic-compatible APIs, GGUF + safetensors + MLX, FP8/NVFP4/MXFP4/Q8/Q4, built-in Studio

    GitHub · Rust · 164 stars · 25 forks · 0 stars in 24 hours · Trending score 0.07

  16. 16
    YuhuaBillChen/justfit-llm

    Run Qwen3.8-27B with its full 256K context on a 24 GB Mac. Long-context local LLM serving for Apple silicon, built on Inco AI's Splash. OpenAI-compatible API.

    GitHub · C++ · 7 stars · 0 forks · 0 stars in 24 hours · Trending score 0.04

  17. 17
    benmaster82/picchio

    A streaming Mixture-of-Experts (MoE) inference engine written in pure C, that runs models larger than your RAM on ordinary consumer hardware - MiniMax-M2 (230B) - GPT-OSS (20B and 120B) and Qwen3-MoE

    GitHub · C · 11 stars · 3 forks · 0 stars in 24 hours · Trending score 0.04

  18. 18
    cl-vv-h/SGLang_Tutor

    Bilingual SGLang tutorials and source-code walkthroughs for LLM inference, AI infrastructure, scheduling, KV cache, attention, Ascend NPU, and kernel optimization. | SGLang 中英双语教程与源码解析

    GitHub · Python · 8 stars · 0 forks · 0 stars in 24 hours · Trending score 0.02

  19. 19
    amitshekhariitbhu/llm-inference-engineering

    Learn LLM Inference Engineering step by step - from KV cache, PagedAttention, and continuous batching to vLLM, SGLang, and GPUs.

    GitHub · Markdown · 405 stars · 50 forks · 0 stars in 24 hours · Trending score 0.02

  20. 20
    pleasedodisturb/awesome-llm-token-optimization

    A curated list of strategies, tools, papers, and resources for reducing LLM token costs and improving efficiency in production.

    GitHub · Shell · 119 stars · 39 forks · 0 stars in 24 hours · Trending score 0.01

  21. 21
    Scottcjn/ram-coffers

    LLM infrastructure cost reduction via NUMA-aware weight banking: 147 t/s (8.8x stock llama.cpp) on refurbished enterprise POWER8. Self-hosted inference, no cloud APIs. Part of the Proof of Physical AI stack.

    GitHub · Python · 173 stars · 59 forks · 0 stars in 24 hours · Trending score 0.01

  22. 22
    brontoguana/krasis

    Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware

    GitHub · C++ · 530 stars · 33 forks · 0 stars in 24 hours · Trending score 0.00

  23. 23
    tanavc1/local-llm-autotune

    Zero-config local LLM optimization for Ollama, LM Studio, and Apple Silicon MLX. Reduces TTFT by 40%, wall time for local agents by 46%, and RAM usage by 3x.

    GitHub · Python · 37 stars · 3 forks · 0 stars in 24 hours · Trending score 0.00

  24. 24
    htobty/ReadyLLM

    Deploy, monitor, and tune LLM through a visual interface — plus image / video generation on top of ComfyUI.

    GitHub · Python · 100 stars · 12 forks · 0 stars in 24 hours · Trending score 0.00