kv-cache repositories

Trending repositories tagged kv-cache, ranked by trending score. Updated daily.

  1. 1
    bojieli/ai-infra-book

    《深入理解 AI Infra:量化分析与系统设计》(李博杰 著)开源书稿:从硬件约束和模型架构出发,量化推导 LLM 推理与训练系统设计。含全书正文、PDF、配套计算工具与实验

    GitHub · Python · 8,154 stars · 666 forks · +93 stars in 24 hours · Trending score 3.93

  2. 2
    datawhalechina/zero-to-sglang

    Official SGLang x Datawhale course on LLM inference: understand inference, build a mini-sglang from scratch, then read the real SGLang source and land your first PR. Available in English and Chinese.

    GitHub · Python · 2,295 stars · 188 forks · +103 stars in 24 hours · Trending score 3.69

  3. 3
    LMCache/LMCache

    LMCache: Supercharge Your LLM with the Fastest KV Cache Layer

    GitHub · Python · 12,398 stars · 2,174 forks · +18 stars in 24 hours · Trending score 2.63

  4. 4
    loxilb-io/loxilb-inference-gateway

    Inference-aware L4/L7 load balancer for LLM serving fleets, forked from loxilb

    GitHub · Go · 71 stars · 14 forks · +18 stars in 24 hours · Trending score 2.62

  5. 5
    OmniJev/Flide

    🔥🔥 The extremely fast inference engine for decision model

    GitHub · Python · 546 stars · 37 forks · +9 stars in 24 hours · Trending score 2.36

  6. 6
    zed123214/kitagent

    KitAgent:面向算子开发与性能优化的 Coding Agent。以 LangGraph 状态机编排开发、验证与迭代,集成工具执行、人工审批、会话恢复、Skills / MCP 扩展与 Kubernetes 沙箱。

    GitHub · Python · 102 stars · 9 forks · +14 stars in 24 hours · Trending score 2.00

  7. 7
    bertholomus/spark-attic

    More memory for CUDA on an NVIDIA GB10 (DGX Spark, ASUS Ascent GX10): lend the unused display carveout to vLLM, SGLang and TensorFold as KV cache

    GitHub · Python · 34 stars · 1 forks · +7 stars in 24 hours · Trending score 1.81

  8. 8
    huawei-csl/KVarN

    KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Calibration-free, one flag.

    GitHub · Python · 521 stars · 38 forks · 0 stars in 24 hours · Trending score 1.27

  9. 9
    tuan3w/qwen-warp

    Fast Qwen3.8-27B for one RTX 3090. 200k context, 100+ tok/s. Custom CUDA decode engine, OpenAI-compatible API.

    GitHub · Python · 49 stars · 6 forks · +3 stars in 24 hours · Trending score 1.16

  10. 10
    Dami-wq-k03/nano-lmcache

    📚 An educational, step-by-step implementation of LLM KV cache reuse — learn prefix caching, chunk keys, storage & eviction, from pure Python to GPU tensors and vLLM integration. Inspired by LMCache.

    GitHub · Python · 43 stars · 13 forks · +1 stars in 24 hours · Trending score 0.96

  11. 11
    AI-Efficiency/Awesome-Model-Quantization

    A curated collection of papers, benchmarks, surveys, and tools for model quantization, covering low-bit networks, LLMs, multimodal and generative models, vector and lattice quantization, and efficient deployment.

    GitHub · 2,470 stars · 246 forks · 0 stars in 24 hours · Trending score 0.86

  12. 12
    OrbitInfra/OrbitKV

    OrbitKV is a High Performance KV Cache System for LLM Inference.

    GitHub · Rust · 284 stars · 19 forks · +1 stars in 24 hours · Trending score 0.82

  13. 13
    anthony-chaudhary/fak

    Agentic Runtime

    GitHub · Go · 50 stars · 16 forks · +1 stars in 24 hours · Trending score 0.78

  14. 14
    NVIDIA/kvpress

    LLM KV cache compression made easy

    GitHub · Python · 1,255 stars · 201 forks · 0 stars in 24 hours · Trending score 0.76

  15. 15
    Wenaixi/dsh-ponytail

    DeepSeek Harness plugin: DietrichGebert/ponytail lazy senior mode & 7-rung ladder port, 6 skills with bilingual descriptions & per-skill toggles, zero tools, zero-cache-miss | DeepSeek Harness 插件:DietrichGebert/ponytail 懒人 senior 模式与七阶梯子完美移植,6 个技能描述双语自由切换,各个技能自由开关,零 tool 注册,全场景零缓存破坏

    GitHub · JavaScript · 46 stars · 3 forks · 0 stars in 24 hours · Trending score 0.75

  16. 16
    Lowaltitudeflight/local-llm-tuning-guide

    A decision library for running local LLMs on multi-GPU machines without NVLink: parallelism, communication, quantization, speculative decoding, KV cache. Every conclusion carries its evidence grade and reproducible conditions.

    GitHub · Python · 105 stars · 0 forks · 0 stars in 24 hours · Trending score 0.72

  17. 17
    lastreload/mnemonic-proxy

    Context management proxy for AI coding agents: verbatim archive, on-demand recall, and saved engine state on supported backends.

    GitHub · Python · 72 stars · 7 forks · 0 stars in 24 hours · Trending score 0.62

  18. 18
    syv-ai/HyperQwen

    Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.

    GitHub · Python · 2,696 stars · 340 forks · 0 stars in 24 hours · Trending score 0.61

  19. 19
    joris-sta/llama-cpp-pascal

    Pre-tensor-core (Pascal GTX 1070) inference optimizations for llama.cpp: native Turbo3 compressed-KV flash attention, PTQ1 ternary DP4A matmuls, chunked gated-delta-net, TurboQuant/VBR KV on 80K-262K contexts

    GitHub · Cuda · 22 stars · 2 forks · 0 stars in 24 hours · Trending score 0.60

  20. 20
    pegainfer-project/pegainfer

    Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

    GitHub · Rust · 745 stars · 121 forks · 0 stars in 24 hours · Trending score 0.57

  21. 21
    aivrar/vllm-windows-build

    Native Windows vLLM 0.33.0 (Python 3.13/CUDA 13.0), plus a Python 3.14/CUDA 13.2 build; older versions on the Releases page. PyTorch 2.14, FlashAttention/Rust, OpenAI serving, 10 KV formats, Multi-TurboQuant, experimental CPU/NVMe KV offload. No WSL or Docker.

    GitHub · Python · 68 stars · 8 forks · 0 stars in 24 hours · Trending score 0.54

  22. 22
    shootthesound/torch-nvenc-compress

    Use the GPU's idle NVENC silicon to multiply the effective bandwidth of every PCIe wire on the board

    GitHub · Python · 32 stars · 5 forks · 0 stars in 24 hours · Trending score 0.42

  23. 23
    LatentPort/latentport

    Telepathy for LLMs.

    GitHub · Python · 13 stars · 1 forks · +1 stars in 24 hours · Trending score 0.41

  24. 24
    sauravsingla/MemVanta

    Low-memory C++20 LLM inference runtime for quantized GGUF models on CPU — mmap-backed weights, paged KV cache, Q4/Q8 kernels, and reproducible llama.cpp benchmarks.

    GitHub · C++ · 41 stars · 40 forks · 0 stars in 24 hours · Trending score 0.37