llm-inference repositories
Trending repositories tagged llm-inference, ranked by trending score. Updated daily.
-
1
openvinotoolkit/openvino
OpenVINO™ is an open source toolkit for optimizing and deploying AI inference
GitHub · C++ · 11,881 stars · 3,647 forks · +465 stars in 24 hours · Trending score 4.78
-
2
autonomous-ai/Personal-AI-Computer
Open-source AI computer builds, from 2-GPU desktops to 8-GPU servers: CAD, BOM, BIOS and build guides. Get the case kit with PCIe risers, or ask us for a free fit check.
GitHub · 1,601 stars · 179 forks · +348 stars in 24 hours · Trending score 4.26
-
3
FareedKhan-dev/kimi-k3-in-c
A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.
GitHub · C · 12,530 stars · 1,940 forks · +127 stars in 24 hours · Trending score 4.17
-
4
incoai/splash
A local inference engine for Apple silicon, built around the model.
GitHub · C++ · 2,597 stars · 250 forks · +167 stars in 24 hours · Trending score 4.00
-
5
bojieli/ai-infra-book
《深入理解 AI Infra:量化分析与系统设计》(李博杰 著)开源书稿:从硬件约束和模型架构出发,量化推导 LLM 推理与训练系统设计。含全书正文、PDF、配套计算工具与实验
GitHub · Python · 8,181 stars · 668 forks · +85 stars in 24 hours · Trending score 3.90
-
6
mw00/project-maya
Run GLM-5.3 (744B), DeepSeek-V4.1-Flash (748B), MiniMax-M3.1-Flash (428B), GLM-5.3-Flash (321B) and Hy4-Flash (220B) on your own GPUs - from 8 GB of VRAM, NVIDIA and AMD, one dashboard, OpenAI- and Anthropic-compatible API.
GitHub · Cuda · 3,143 stars · 442 forks · +92 stars in 24 hours · Trending score 3.80
-
7
ashhart/TensorFold
LLM Inference Engine for Metal, CUDA and Vulkan.
GitHub · Zig · 2,367 stars · 551 forks · +30 stars in 24 hours · Trending score 3.60
-
8
nicorobinreal/llmspeed
Feel how fast a local LLM runs before you buy the hardware
GitHub · TypeScript · 200 stars · 8 forks · +116 stars in 24 hours · Trending score 3.48
-
9
wici-ai/inferssd
Run models larger than your RAM or VRAM by streaming MoE experts from SSD. Built on llama.cpp.
GitHub · C++ · 1,278 stars · 115 forks · +44 stars in 24 hours · Trending score 3.44
-
10
datawhalechina/zero-to-sglang
Official SGLang x Datawhale course on LLM inference: understand inference, build a mini-sglang from scratch, then read the real SGLang source and land your first PR. Available in English and Chinese.
GitHub · Python · 2,307 stars · 189 forks · +37 stars in 24 hours · Trending score 3.20
-
11
ray-project/ray
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
GitHub · Python · 44,352 stars · 8,296 forks · +26 stars in 24 hours · Trending score 2.87
-
12
flashinfer-ai/flashinfer
FlashInfer: Kernel Library for LLM Serving
GitHub · Cuda · 6,981 stars · 1,789 forks · +18 stars in 24 hours · Trending score 2.78
-
13
Picom-code/pico-dgx-ds4
DeepSeek V4.1 Flash on 4x DGX Spark: 90/144 tok/s chat/code single-user, 376 tok/s at 16 distinct users, 1M context, images. Fork of blaesus/dsgx10.
GitHub · Cuda · 151 stars · 28 forks · +28 stars in 24 hours · Trending score 2.78
-
14
ai-dynamo/dynamo
A Datacenter Scale Distributed Inference Serving Framework
GitHub · Rust · 8,543 stars · 1,887 forks · +18 stars in 24 hours · Trending score 2.76
-
15
thejarasal780-ship-it/ollama-local-ai-offline
Ollama Local AI Offline: Run Local LLMs Fully Offline on Desktop
GitHub · Python · 152 stars · 0 forks · +120 stars in 24 hours · Trending score 2.66
-
16
bertholomus/deepseek-v4.1-tensorfold-tp2-2xgb10
BertholomusAI recipe: DeepSeek-V4.1-Flash-1017 on 2x NVIDIA DGX Spark (GB10), our own TensorFold TP2 engine. Weights: bertholomus EXL3 2.5bpw quant (MIT). v0.7.5.
GitHub · Python · 73 stars · 7 forks · +33 stars in 24 hours · Trending score 2.64
-
17
gpustack/gpustack
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
GitHub · Python · 6,097 stars · 748 forks · +18 stars in 24 hours · Trending score 2.57
-
18
loxilb-io/loxilb-inference-gateway
Inference-aware L4/L7 load balancer for LLM serving fleets, forked from loxilb
GitHub · Go · 73 stars · 14 forks · +15 stars in 24 hours · Trending score 2.54
-
19
lemonade-sdk/lemonade
Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs. Join our discord: https://discord.gg/5xXzkMu8Zk
GitHub · C++ · 6,263 stars · 571 forks · +15 stars in 24 hours · Trending score 2.48
-
20
AtomicBot-ai/Atomic-Chat
Local AI app and inference engine for agents. Run open-weight LLMs locally — private, 100% offline on your computer. Join our Discord: https://discord.com/invite/8wGSsvmg4V
GitHub · TypeScript · 2,038 stars · 235 forks · +25 stars in 24 hours · Trending score 2.48
-
21
nace-ai/kapitsa
Blazing fast Apple Silicon inference engine optimized for Qwen & Nemotron. OpenAI + Anthropic API compatible. Built by folks from nace.ai
GitHub · C++ · 836 stars · 116 forks · +11 stars in 24 hours · Trending score 2.44
-
22
SemiAnalysisAI/InferenceX
Open Source AI Accelerator Research Platform Standard / 开源推理研究平台
GitHub · Python · 2,074 stars · 386 forks · +13 stars in 24 hours · Trending score 2.43
-
23
senickimoro-art/pinokio-olvara-ai-workspace
A modular desktop workspace for configuring and running local AI models.
GitHub · Dockerfile · 152 stars · 0 forks · +105 stars in 24 hours · Trending score 2.43
-
24
tetherto/qvac
Open-source local AI SDK - run AI on-device with no cloud, no API keys. Supports GGUF, RAG, image, music, and video generation, speech-to-text, P2P inference, and more. Cross-platform: Linux, macOS, Windows, Android, iOS.
GitHub · TypeScript · 804 stars · 168 forks · +17 stars in 24 hours · Trending score 2.40