llm-inference repositories

Trending repositories tagged llm-inference, ranked by trending score. Updated daily.

  1. 1
    openvinotoolkit/openvino

    OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

    GitHub · C++ · 11,881 stars · 3,647 forks · +465 stars in 24 hours · Trending score 4.78

  2. 2
    autonomous-ai/Personal-AI-Computer

    Open-source AI computer builds, from 2-GPU desktops to 8-GPU servers: CAD, BOM, BIOS and build guides. Get the case kit with PCIe risers, or ask us for a free fit check.

    GitHub · 1,601 stars · 179 forks · +348 stars in 24 hours · Trending score 4.26

  3. 3
    FareedKhan-dev/kimi-k3-in-c

    A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.

    GitHub · C · 12,530 stars · 1,940 forks · +127 stars in 24 hours · Trending score 4.17

  4. 4
    incoai/splash

    A local inference engine for Apple silicon, built around the model.

    GitHub · C++ · 2,597 stars · 250 forks · +167 stars in 24 hours · Trending score 4.00

  5. 5
    bojieli/ai-infra-book

    《深入理解 AI Infra:量化分析与系统设计》(李博杰 著)开源书稿:从硬件约束和模型架构出发,量化推导 LLM 推理与训练系统设计。含全书正文、PDF、配套计算工具与实验

    GitHub · Python · 8,181 stars · 668 forks · +85 stars in 24 hours · Trending score 3.90

  6. 6
    mw00/project-maya

    Run GLM-5.3 (744B), DeepSeek-V4.1-Flash (748B), MiniMax-M3.1-Flash (428B), GLM-5.3-Flash (321B) and Hy4-Flash (220B) on your own GPUs - from 8 GB of VRAM, NVIDIA and AMD, one dashboard, OpenAI- and Anthropic-compatible API.

    GitHub · Cuda · 3,143 stars · 442 forks · +92 stars in 24 hours · Trending score 3.80

  7. 7
    ashhart/TensorFold

    LLM Inference Engine for Metal, CUDA and Vulkan.

    GitHub · Zig · 2,367 stars · 551 forks · +30 stars in 24 hours · Trending score 3.60

  8. 8
    nicorobinreal/llmspeed

    Feel how fast a local LLM runs before you buy the hardware

    GitHub · TypeScript · 200 stars · 8 forks · +116 stars in 24 hours · Trending score 3.48

  9. 9
    wici-ai/inferssd

    Run models larger than your RAM or VRAM by streaming MoE experts from SSD. Built on llama.cpp.

    GitHub · C++ · 1,278 stars · 115 forks · +44 stars in 24 hours · Trending score 3.44

  10. 10
    datawhalechina/zero-to-sglang

    Official SGLang x Datawhale course on LLM inference: understand inference, build a mini-sglang from scratch, then read the real SGLang source and land your first PR. Available in English and Chinese.

    GitHub · Python · 2,307 stars · 189 forks · +37 stars in 24 hours · Trending score 3.20

  11. 11
    ray-project/ray

    Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

    GitHub · Python · 44,352 stars · 8,296 forks · +26 stars in 24 hours · Trending score 2.87

  12. 12
    flashinfer-ai/flashinfer

    FlashInfer: Kernel Library for LLM Serving

    GitHub · Cuda · 6,981 stars · 1,789 forks · +18 stars in 24 hours · Trending score 2.78

  13. 13
    Picom-code/pico-dgx-ds4

    DeepSeek V4.1 Flash on 4x DGX Spark: 90/144 tok/s chat/code single-user, 376 tok/s at 16 distinct users, 1M context, images. Fork of blaesus/dsgx10.

    GitHub · Cuda · 151 stars · 28 forks · +28 stars in 24 hours · Trending score 2.78

  14. 14
    ai-dynamo/dynamo

    A Datacenter Scale Distributed Inference Serving Framework

    GitHub · Rust · 8,543 stars · 1,887 forks · +18 stars in 24 hours · Trending score 2.76

  15. 15
    thejarasal780-ship-it/ollama-local-ai-offline

    Ollama Local AI Offline: Run Local LLMs Fully Offline on Desktop

    GitHub · Python · 152 stars · 0 forks · +120 stars in 24 hours · Trending score 2.66

  16. 16
    bertholomus/deepseek-v4.1-tensorfold-tp2-2xgb10

    BertholomusAI recipe: DeepSeek-V4.1-Flash-1017 on 2x NVIDIA DGX Spark (GB10), our own TensorFold TP2 engine. Weights: bertholomus EXL3 2.5bpw quant (MIT). v0.7.5.

    GitHub · Python · 73 stars · 7 forks · +33 stars in 24 hours · Trending score 2.64

  17. 17
    gpustack/gpustack

    A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

    GitHub · Python · 6,097 stars · 748 forks · +18 stars in 24 hours · Trending score 2.57

  18. 18
    loxilb-io/loxilb-inference-gateway

    Inference-aware L4/L7 load balancer for LLM serving fleets, forked from loxilb

    GitHub · Go · 73 stars · 14 forks · +15 stars in 24 hours · Trending score 2.54

  19. 19
    lemonade-sdk/lemonade

    Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs. Join our discord: https://discord.gg/5xXzkMu8Zk

    GitHub · C++ · 6,263 stars · 571 forks · +15 stars in 24 hours · Trending score 2.48

  20. 20
    AtomicBot-ai/Atomic-Chat

    Local AI app and inference engine for agents. Run open-weight LLMs locally — private, 100% offline on your computer. Join our Discord: https://discord.com/invite/8wGSsvmg4V

    GitHub · TypeScript · 2,038 stars · 235 forks · +25 stars in 24 hours · Trending score 2.48

  21. 21
    nace-ai/kapitsa

    Blazing fast Apple Silicon inference engine optimized for Qwen & Nemotron. OpenAI + Anthropic API compatible. Built by folks from nace.ai

    GitHub · C++ · 836 stars · 116 forks · +11 stars in 24 hours · Trending score 2.44

  22. 22
    SemiAnalysisAI/InferenceX

    Open Source AI Accelerator Research Platform Standard / 开源推理研究平台

    GitHub · Python · 2,074 stars · 386 forks · +13 stars in 24 hours · Trending score 2.43

  23. 23
    senickimoro-art/pinokio-olvara-ai-workspace

    A modular desktop workspace for configuring and running local AI models.

    GitHub · Dockerfile · 152 stars · 0 forks · +105 stars in 24 hours · Trending score 2.43

  24. 24
    tetherto/qvac

    Open-source local AI SDK - run AI on-device with no cloud, no API keys. Supports GGUF, RAG, image, music, and video generation, speech-to-text, P2P inference, and more. Cross-platform: Linux, macOS, Windows, Android, iOS.

    GitHub · TypeScript · 804 stars · 168 forks · +17 stars in 24 hours · Trending score 2.40