inference repositories

Trending repositories tagged inference, ranked by trending score. Updated daily.

  1. 1
    openvinotoolkit/openvino

    OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

    GitHub · C++ · 11,649 stars · 3,629 forks · +348 stars in 24 hours · Trending score 4.53

  2. 2
    FlashML-org/FreeVideo

    Make videos on the computer you already own. FreeVideo runs MiniMax H3 in as little as 8 GB of VRAM and 16 GB of RAM, and adapts its acceleration path to your hardware.

    GitHub · Python · 3,578 stars · 487 forks · +194 stars in 24 hours · Trending score 4.29

  3. 3
    vllm-project/vllm

    A high-throughput and memory-efficient inference and serving engine for LLMs

    GitHub · Python · 97,255 stars · 25,073 forks · +109 stars in 24 hours · Trending score 4.26

  4. 4
    ggml-org/whisper.cpp

    Port of OpenAI's Whisper model in C/C++

    GitHub · C++ · 55,359 stars · 6,336 forks · +70 stars in 24 hours · Trending score 3.62

  5. 5
    FlashML-org/FreeToken

    FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.

    GitHub · Python · 17,331 stars · 1,721 forks · +44 stars in 24 hours · Trending score 3.43

  6. 6
    vllm-project/vllm-omni

    A framework for efficient model inference with omni-modality models

    GitHub · Python · 7,856 stars · 2,162 forks · +36 stars in 24 hours · Trending score 3.29

  7. 7
    Veloresearch/kinetic

    The open engine for running AI on your own computer. Fast, exact, open source.

    GitHub · Rust · 202 stars · 20 forks · +62 stars in 24 hours · Trending score 3.17

  8. 8
    RunanywhereAI/trnrt

    The open LLM inference engine for AWS Trainium: NKI megakernels, 1M-token context, OpenAI-compatible API. By RunAnywhere.

    GitHub · Python · 1,639 stars · 24 forks · +31 stars in 24 hours · Trending score 3.16

  9. 9
    Helldez/BigMoeOnEdge

    Run MoE models bigger than your RAM. Frontier-size MoE on a phone, laptop, edge. Lossless, on stock llama.cpp. An adaptive streaming optimizer. Any hardware, any OS, any model, any quantization, behind a local OpenAI-compatible server.

    GitHub · C++ · 1,033 stars · 107 forks · +23 stars in 24 hours · Trending score 3.07

  10. 10
    vllm-project/vllm-ascend

    Community maintained hardware plugin for vLLM on Huawei Ascend

    GitHub · Python · 3,124 stars · 2,788 forks · +24 stars in 24 hours · Trending score 2.96

  11. 11
    RustingStudio/RustingStrata

    Pure-Rust inference engine for Qwen GGUF models (Qwen3 to Qwen3.8, dense and MoE up to 770B+). CUDA and Vulkan (NVIDIA, AMD), hybrid CPU+GPU experts, MTP speculative decoding, OpenAI and Anthropic compatible server. Up to ~1.7x llama.cpp decode on a 12 GB RTX 3060.

    GitHub · Rust · 291 stars · 36 forks · +31 stars in 24 hours · Trending score 2.90

  12. 12
    sgl-project/sglang-omni

    High Performance Serving Runtime For Interactive Super Intelligence System

    GitHub · Python · 1,827 stars · 762 forks · +15 stars in 24 hours · Trending score 2.83

  13. 13
    llm-d/llm-d

    State of the art inference performance with modern accelerators on Kubernetes

    GitHub · Python · 5,319 stars · 944 forks · +25 stars in 24 hours · Trending score 2.83

  14. 14
    kvcache-ai/Mooncake

    Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

    GitHub · C++ · 7,088 stars · 1,507 forks · +22 stars in 24 hours · Trending score 2.80

  15. 15
    sybil-solutions/inference-optimisation-skill

    Agent skill for hill-climbing local LLM serving speed (decode, speculative decoding, prefill, KV, startup, TTFT) on any hardware and quantization

    GitHub · Python · 1,284 stars · 114 forks · +372 stars in 24 hours · Trending score 2.77

  16. 16
    stas00/ml-engineering

    Machine Learning Engineering Open Book

    GitHub · Python · 19,732 stars · 1,304 forks · +26 stars in 24 hours · Trending score 2.67

  17. 17
    vllm-project/semantic-router

    An open, programmable decision layer for models and compute.

    GitHub · Go · 6,512 stars · 1,140 forks · +17 stars in 24 hours · Trending score 2.66

  18. 18
    LMCache/LMCache

    LMCache: Supercharge Your LLM with the Fastest KV Cache Layer

    GitHub · Python · 12,398 stars · 2,174 forks · +18 stars in 24 hours · Trending score 2.63

  19. 19
    DevDock-AI/Speedy

    Qwen3.8-Flash-Next (125B MoE) up to 1M context on one or two RTX cards (tuned on 3080 10gb). A measured, tuned and benchmarked fork of Strata.

    GitHub · C++ · 125 stars · 8 forks · +26 stars in 24 hours · Trending score 2.59

  20. 20
    aryamandarda/tamserve

    Live, multi-camera object tracking with SAM3, SAM 2 and EdgeTAM, on one GPU or many. One model per GPU serves every camera: faster than the original code, about 4x less GPU memory, the same accuracy.

    GitHub · Python · 255 stars · 14 forks · +163 stars in 24 hours · Trending score 2.55

  21. 21
    google-ai-edge/mediapipe

    Cross-platform, customizable ML solutions for live and streaming media.

    GitHub · C++ · 37,753 stars · 6,236 forks · +13 stars in 24 hours · Trending score 2.54

  22. 22
    chigwell/llm7.io

    LLM7.io offers a single API gateway that connects you to a wide array of leading AI models from various providers.

    GitHub · TypeScript · 265 stars · 14 forks · +25 stars in 24 hours · Trending score 2.47

  23. 23
    SemiAnalysisAI/InferenceX

    Open Source AI Accelerator Research Platform Standard / 开源推理研究平台

    GitHub · Python · 2,069 stars · 382 forks · +15 stars in 24 hours · Trending score 2.47

  24. 24
    gpustack/gpustack

    A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

    GitHub · Python · 6,086 stars · 746 forks · +11 stars in 24 hours · Trending score 2.42