kv-cache repositories
Trending repositories tagged kv-cache, ranked by trending score. Updated daily.
-
1
bojieli/ai-infra-book
《深入理解 AI Infra:量化分析与系统设计》(李博杰 著)开源书稿:从硬件约束和模型架构出发,量化推导 LLM 推理与训练系统设计。含全书正文、PDF、配套计算工具与实验
GitHub · Python · 8,154 stars · 666 forks · +93 stars in 24 hours · Trending score 3.93
-
2
datawhalechina/zero-to-sglang
Official SGLang x Datawhale course on LLM inference: understand inference, build a mini-sglang from scratch, then read the real SGLang source and land your first PR. Available in English and Chinese.
GitHub · Python · 2,295 stars · 188 forks · +103 stars in 24 hours · Trending score 3.69
-
3
LMCache/LMCache
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
GitHub · Python · 12,398 stars · 2,174 forks · +18 stars in 24 hours · Trending score 2.63
-
4
loxilb-io/loxilb-inference-gateway
Inference-aware L4/L7 load balancer for LLM serving fleets, forked from loxilb
GitHub · Go · 71 stars · 14 forks · +18 stars in 24 hours · Trending score 2.62
-
5
OmniJev/Flide
🔥🔥 The extremely fast inference engine for decision model
GitHub · Python · 546 stars · 37 forks · +9 stars in 24 hours · Trending score 2.36
-
6
zed123214/kitagent
KitAgent:面向算子开发与性能优化的 Coding Agent。以 LangGraph 状态机编排开发、验证与迭代,集成工具执行、人工审批、会话恢复、Skills / MCP 扩展与 Kubernetes 沙箱。
GitHub · Python · 102 stars · 9 forks · +14 stars in 24 hours · Trending score 2.00
-
7
bertholomus/spark-attic
More memory for CUDA on an NVIDIA GB10 (DGX Spark, ASUS Ascent GX10): lend the unused display carveout to vLLM, SGLang and TensorFold as KV cache
GitHub · Python · 34 stars · 1 forks · +7 stars in 24 hours · Trending score 1.81
-
8
huawei-csl/KVarN
KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Calibration-free, one flag.
GitHub · Python · 521 stars · 38 forks · 0 stars in 24 hours · Trending score 1.27
-
9
tuan3w/qwen-warp
Fast Qwen3.8-27B for one RTX 3090. 200k context, 100+ tok/s. Custom CUDA decode engine, OpenAI-compatible API.
GitHub · Python · 49 stars · 6 forks · +3 stars in 24 hours · Trending score 1.16
-
10
Dami-wq-k03/nano-lmcache
📚 An educational, step-by-step implementation of LLM KV cache reuse — learn prefix caching, chunk keys, storage & eviction, from pure Python to GPU tensors and vLLM integration. Inspired by LMCache.
GitHub · Python · 43 stars · 13 forks · +1 stars in 24 hours · Trending score 0.96
-
11
AI-Efficiency/Awesome-Model-Quantization
A curated collection of papers, benchmarks, surveys, and tools for model quantization, covering low-bit networks, LLMs, multimodal and generative models, vector and lattice quantization, and efficient deployment.
GitHub · 2,470 stars · 246 forks · 0 stars in 24 hours · Trending score 0.86
-
12
OrbitInfra/OrbitKV
OrbitKV is a High Performance KV Cache System for LLM Inference.
GitHub · Rust · 284 stars · 19 forks · +1 stars in 24 hours · Trending score 0.82
-
13
anthony-chaudhary/fak
Agentic Runtime
GitHub · Go · 50 stars · 16 forks · +1 stars in 24 hours · Trending score 0.78
-
14
NVIDIA/kvpress
LLM KV cache compression made easy
GitHub · Python · 1,255 stars · 201 forks · 0 stars in 24 hours · Trending score 0.76
-
15
Wenaixi/dsh-ponytail
DeepSeek Harness plugin: DietrichGebert/ponytail lazy senior mode & 7-rung ladder port, 6 skills with bilingual descriptions & per-skill toggles, zero tools, zero-cache-miss | DeepSeek Harness 插件:DietrichGebert/ponytail 懒人 senior 模式与七阶梯子完美移植,6 个技能描述双语自由切换,各个技能自由开关,零 tool 注册,全场景零缓存破坏
GitHub · JavaScript · 46 stars · 3 forks · 0 stars in 24 hours · Trending score 0.75
-
16
Lowaltitudeflight/local-llm-tuning-guide
A decision library for running local LLMs on multi-GPU machines without NVLink: parallelism, communication, quantization, speculative decoding, KV cache. Every conclusion carries its evidence grade and reproducible conditions.
GitHub · Python · 105 stars · 0 forks · 0 stars in 24 hours · Trending score 0.72
-
17
lastreload/mnemonic-proxy
Context management proxy for AI coding agents: verbatim archive, on-demand recall, and saved engine state on supported backends.
GitHub · Python · 72 stars · 7 forks · 0 stars in 24 hours · Trending score 0.62
-
18
syv-ai/HyperQwen
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
GitHub · Python · 2,696 stars · 340 forks · 0 stars in 24 hours · Trending score 0.61
-
19
joris-sta/llama-cpp-pascal
Pre-tensor-core (Pascal GTX 1070) inference optimizations for llama.cpp: native Turbo3 compressed-KV flash attention, PTQ1 ternary DP4A matmuls, chunked gated-delta-net, TurboQuant/VBR KV on 80K-262K contexts
GitHub · Cuda · 22 stars · 2 forks · 0 stars in 24 hours · Trending score 0.60
-
20
pegainfer-project/pegainfer
Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2
GitHub · Rust · 745 stars · 121 forks · 0 stars in 24 hours · Trending score 0.57
-
21
aivrar/vllm-windows-build
Native Windows vLLM 0.33.0 (Python 3.13/CUDA 13.0), plus a Python 3.14/CUDA 13.2 build; older versions on the Releases page. PyTorch 2.14, FlashAttention/Rust, OpenAI serving, 10 KV formats, Multi-TurboQuant, experimental CPU/NVMe KV offload. No WSL or Docker.
GitHub · Python · 68 stars · 8 forks · 0 stars in 24 hours · Trending score 0.54
-
22
shootthesound/torch-nvenc-compress
Use the GPU's idle NVENC silicon to multiply the effective bandwidth of every PCIe wire on the board
GitHub · Python · 32 stars · 5 forks · 0 stars in 24 hours · Trending score 0.42
-
23
LatentPort/latentport
Telepathy for LLMs.
GitHub · Python · 13 stars · 1 forks · +1 stars in 24 hours · Trending score 0.41
-
24
sauravsingla/MemVanta
Low-memory C++20 LLM inference runtime for quantized GGUF models on CPU — mmap-backed weights, paged KV cache, Q4/Q8 kernels, and reproducible llama.cpp benchmarks.
GitHub · C++ · 41 stars · 40 forks · 0 stars in 24 hours · Trending score 0.37