distributed-inference repositories
Trending repositories tagged distributed-inference, ranked by trending score. Updated daily.
-
1
sgl-project/sglang-omni
High Performance Serving Runtime For Interactive Super Intelligence System
GitHub · Python · 1,827 stars · 762 forks · +15 stars in 24 hours · Trending score 2.83
-
2
llm-d/llm-d
State of the art inference performance with modern accelerators on Kubernetes
GitHub · Python · 5,319 stars · 944 forks · +25 stars in 24 hours · Trending score 2.83
-
3
flashinfer-ai/flashinfer
FlashInfer: Kernel Library for LLM Serving
GitHub · Cuda · 6,961 stars · 1,773 forks · +21 stars in 24 hours · Trending score 2.81
-
4
gpustack/gpustack
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
GitHub · Python · 6,086 stars · 746 forks · +11 stars in 24 hours · Trending score 2.42
-
5
christopherowen/spark-ds41f
Reproducible Docker/vLLM deployment, tuning, and benchmarks for DeepSeek V4.1 Flash on a switchless three-node or four-node DGX Spark fabric.
GitHub · Python · 235 stars · 58 forks · +6 stars in 24 hours · Trending score 2.17
-
6
hyukudan/strix-halo-engine-lab
Exact LLM inference engine written from scratch for AMD Strix Halo (gfx1151, ROCm/HIP). Qwen3.8-Flash-Next on one node or two replicas; DeepSeek V4.1 Flash across two nodes over USB4 with an optional RTX PRO. Speculative decoding bit-identical to greedy, up to 255 tok/s on coding chat. MiniMax and GLM next.
GitHub · C++ · 37 stars · 6 forks · +4 stars in 24 hours · Trending score 1.48
-
7
enapt/SwarmLLM
Free, open-source app that runs AI chat models on your own computer — and lets computers team up over the internet to run models too big for one machine. No account, no crypto. OpenAI- and Anthropic-compatible API.
GitHub · Rust · 90 stars · 13 forks · +2 stars in 24 hours · Trending score 1.05
-
8
JustVugg/lumabri
Run large AI models across your computers and servers. Powered by Colibri. Pure C.
GitHub · C · 228 stars · 31 forks · 0 stars in 24 hours · Trending score 0.88
-
9
awslabs/awsome-distributed-ai
Best practices, reference architectures, and examples for distributed AI training and inference on AWS.
GitHub · Shell · 499 stars · 216 forks · +1 stars in 24 hours · Trending score 0.77
-
10
dakotacookson/Llama-RPC-Farm
Web UI to pool your home machines into one llama.cpp cluster: multi-machine RPC GPU pooling + on-demand multi-model serving via llama-swap
GitHub · Python · 11 stars · 2 forks · 0 stars in 24 hours · Trending score 0.37
-
11
joshuaswarren/omarchy-cluster
Run one local MLX model across Apple Silicon Macs on macOS and on Omarchy Linux, served as one OpenAI-compatible endpoint
GitHub · Python · 25 stars · 1 forks · 0 stars in 24 hours · Trending score 0.35
-
12
Ninnix/q36
Qwen3.8-27B and Qwen3.6-35B-A3B local inference engine for Vulkan and Metal
GitHub · C · 83 stars · 9 forks · 0 stars in 24 hours · Trending score 0.27
-
13
yeixio/toskar-core
Run AI models like Llama, Qwen, and Gemma on every computer you own, and manage them for everyone on your network. Local first, with OpenAI- and Anthropic-compatible APIs, people and roles, and MCP.
GitHub · Go · 5 stars · 6 forks · 0 stars in 24 hours · Trending score 0.09
-
14
shootthesound/comfyui-mesh
Split FLUX.2 and LTX 2.3 across two GPUs (LAN or same-machine) — NVENC compresses activations live on the wire. Icarus (ComfyUI node) + Daedalus (back-half server).
GitHub · Python · 125 stars · 14 forks · 0 stars in 24 hours · Trending score 0.00
-
15
labscommunity/cascadia
Run any model on Intel silicon
GitHub · Rust · 52 stars · 1 forks · 0 stars in 24 hours · Trending score 0.00
-
16
b-ostrov/MelonDMA
Experimental PCIDriverKit driver for Mellanox ConnectX RDMA cards on macOS — RoCEv2 clustering between Mac Studios, no kernel extension required.
GitHub · C++ · 10 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00
-
17
yunwei37/dgx-spark-4-ring-no-switch
Reproducible LLM inference on four NVIDIA DGX Spark (GB10) nodes wired as a direct switchless ConnectX ring — serving profiles, ring launchers, pinned ARM64 runtime images and measured results, newest tested model first.
GitHub · Python · 20 stars · 4 forks · 0 stars in 24 hours · Trending score 0.00
-
18
trademav/ramdeck-core-public
GitHub · Python · 57 stars · 10 forks · 0 stars in 24 hours · Trending score 0.00
-
19
Nehanth/pooled
Peer-to-peer LLM inference in the browser: pool your devices to run big open models, for chat and coding agents.
GitHub · JavaScript · 624 stars · 95 forks · 0 stars in 24 hours · Trending score 0.00
-
20
mr-tbot/MeshCompute
Decentralized peer-to-peer AI compute network — run one model across many machines over the open internet. Standalone GPU nodes (self-installing, no LM Studio/Ollama), QUIC + NAT hole-punch, BitTorrent-style model distribution, topology-aware load-balancing scheduler, and a blockchain-style verifiable credit ledger. Phase 1 POC.
GitHub · Python · 5 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00
-
21
christopherowen/dgx-spark-networking
Switchless RoCE collectives, NCCL profiles and fabric tooling for DGX Spark inference recipes
GitHub · Python · 30 stars · 3 forks · 0 stars in 24 hours · Trending score 0.00
-
22
Deen-Media/dgx-monarch
Distributed image and video generation in ComfyUI for two NVIDIA DGX Sparks.
GitHub · Python · 32 stars · 3 forks · 0 stars in 24 hours · Trending score 0.00
-
23
jyatesdotdev/strix-rdma
General RDMA transport between two Strix Halo hosts over Thunderbolt/USB4 NHI DMA rings: a native software-verbs provider with a zero-copy data plane, plus the USB4STREAM backport it builds on
GitHub · C · 5 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00
-
24
Dharshaneshwaran/hyperpod
Pools consumer laptops into one local runtime for Meta Llama models. Measures the free VRAM each machine reports, splits the model across CUDA and Metal devices through llama.cpp RPC, and rebuilds the pipeline when a worker drops out.
GitHub · Go · 56 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00