distributed-inference repositories

Trending repositories tagged distributed-inference, ranked by trending score. Updated daily.

  1. 1
    sgl-project/sglang-omni

    High Performance Serving Runtime For Interactive Super Intelligence System

    GitHub · Python · 1,827 stars · 762 forks · +15 stars in 24 hours · Trending score 2.83

  2. 2
    llm-d/llm-d

    State of the art inference performance with modern accelerators on Kubernetes

    GitHub · Python · 5,319 stars · 944 forks · +25 stars in 24 hours · Trending score 2.83

  3. 3
    flashinfer-ai/flashinfer

    FlashInfer: Kernel Library for LLM Serving

    GitHub · Cuda · 6,961 stars · 1,773 forks · +21 stars in 24 hours · Trending score 2.81

  4. 4
    gpustack/gpustack

    A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

    GitHub · Python · 6,086 stars · 746 forks · +11 stars in 24 hours · Trending score 2.42

  5. 5
    christopherowen/spark-ds41f

    Reproducible Docker/vLLM deployment, tuning, and benchmarks for DeepSeek V4.1 Flash on a switchless three-node or four-node DGX Spark fabric.

    GitHub · Python · 235 stars · 58 forks · +6 stars in 24 hours · Trending score 2.17

  6. 6
    hyukudan/strix-halo-engine-lab

    Exact LLM inference engine written from scratch for AMD Strix Halo (gfx1151, ROCm/HIP). Qwen3.8-Flash-Next on one node or two replicas; DeepSeek V4.1 Flash across two nodes over USB4 with an optional RTX PRO. Speculative decoding bit-identical to greedy, up to 255 tok/s on coding chat. MiniMax and GLM next.

    GitHub · C++ · 37 stars · 6 forks · +4 stars in 24 hours · Trending score 1.48

  7. 7
    enapt/SwarmLLM

    Free, open-source app that runs AI chat models on your own computer — and lets computers team up over the internet to run models too big for one machine. No account, no crypto. OpenAI- and Anthropic-compatible API.

    GitHub · Rust · 90 stars · 13 forks · +2 stars in 24 hours · Trending score 1.05

  8. 8
    JustVugg/lumabri

    Run large AI models across your computers and servers. Powered by Colibri. Pure C.

    GitHub · C · 228 stars · 31 forks · 0 stars in 24 hours · Trending score 0.88

  9. 9
    awslabs/awsome-distributed-ai

    Best practices, reference architectures, and examples for distributed AI training and inference on AWS.

    GitHub · Shell · 499 stars · 216 forks · +1 stars in 24 hours · Trending score 0.77

  10. 10
    dakotacookson/Llama-RPC-Farm

    Web UI to pool your home machines into one llama.cpp cluster: multi-machine RPC GPU pooling + on-demand multi-model serving via llama-swap

    GitHub · Python · 11 stars · 2 forks · 0 stars in 24 hours · Trending score 0.37

  11. 11
    joshuaswarren/omarchy-cluster

    Run one local MLX model across Apple Silicon Macs on macOS and on Omarchy Linux, served as one OpenAI-compatible endpoint

    GitHub · Python · 25 stars · 1 forks · 0 stars in 24 hours · Trending score 0.35

  12. 12
    Ninnix/q36

    Qwen3.8-27B and Qwen3.6-35B-A3B local inference engine for Vulkan and Metal

    GitHub · C · 83 stars · 9 forks · 0 stars in 24 hours · Trending score 0.27

  13. 13
    yeixio/toskar-core

    Run AI models like Llama, Qwen, and Gemma on every computer you own, and manage them for everyone on your network. Local first, with OpenAI- and Anthropic-compatible APIs, people and roles, and MCP.

    GitHub · Go · 5 stars · 6 forks · 0 stars in 24 hours · Trending score 0.09

  14. 14
    shootthesound/comfyui-mesh

    Split FLUX.2 and LTX 2.3 across two GPUs (LAN or same-machine) — NVENC compresses activations live on the wire. Icarus (ComfyUI node) + Daedalus (back-half server).

    GitHub · Python · 125 stars · 14 forks · 0 stars in 24 hours · Trending score 0.00

  15. 15
    labscommunity/cascadia

    Run any model on Intel silicon

    GitHub · Rust · 52 stars · 1 forks · 0 stars in 24 hours · Trending score 0.00

  16. 16
    b-ostrov/MelonDMA

    Experimental PCIDriverKit driver for Mellanox ConnectX RDMA cards on macOS — RoCEv2 clustering between Mac Studios, no kernel extension required.

    GitHub · C++ · 10 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00

  17. 17
    yunwei37/dgx-spark-4-ring-no-switch

    Reproducible LLM inference on four NVIDIA DGX Spark (GB10) nodes wired as a direct switchless ConnectX ring — serving profiles, ring launchers, pinned ARM64 runtime images and measured results, newest tested model first.

    GitHub · Python · 20 stars · 4 forks · 0 stars in 24 hours · Trending score 0.00

  18. 18
    trademav/ramdeck-core-public

    GitHub · Python · 57 stars · 10 forks · 0 stars in 24 hours · Trending score 0.00

  19. 19
    Nehanth/pooled

    Peer-to-peer LLM inference in the browser: pool your devices to run big open models, for chat and coding agents.

    GitHub · JavaScript · 624 stars · 95 forks · 0 stars in 24 hours · Trending score 0.00

  20. 20
    mr-tbot/MeshCompute

    Decentralized peer-to-peer AI compute network — run one model across many machines over the open internet. Standalone GPU nodes (self-installing, no LM Studio/Ollama), QUIC + NAT hole-punch, BitTorrent-style model distribution, topology-aware load-balancing scheduler, and a blockchain-style verifiable credit ledger. Phase 1 POC.

    GitHub · Python · 5 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00

  21. 21
    christopherowen/dgx-spark-networking

    Switchless RoCE collectives, NCCL profiles and fabric tooling for DGX Spark inference recipes

    GitHub · Python · 30 stars · 3 forks · 0 stars in 24 hours · Trending score 0.00

  22. 22
    Deen-Media/dgx-monarch

    Distributed image and video generation in ComfyUI for two NVIDIA DGX Sparks.

    GitHub · Python · 32 stars · 3 forks · 0 stars in 24 hours · Trending score 0.00

  23. 23
    jyatesdotdev/strix-rdma

    General RDMA transport between two Strix Halo hosts over Thunderbolt/USB4 NHI DMA rings: a native software-verbs provider with a zero-copy data plane, plus the USB4STREAM backport it builds on

    GitHub · C · 5 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00

  24. 24
    Dharshaneshwaran/hyperpod

    Pools consumer laptops into one local runtime for Meta Llama models. Measures the free VRAM each machine reports, splits the model across CUDA and Metal devices through llama.cpp RPC, and rebuilds the pipeline when a worker drops out.

    GitHub · Go · 56 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00