llm-serving repositories

Trending repositories tagged llm-serving, ranked by trending score. Updated daily.

  1. 1
    vllm-project/vllm

    A high-throughput and memory-efficient inference and serving engine for LLMs

    GitHub · Python · 97,255 stars · 25,073 forks · +109 stars in 24 hours · Trending score 4.26

  2. 2
    bojieli/ai-infra-book

    《深入理解 AI Infra:量化分析与系统设计》(李博杰 著)开源书稿:从硬件约束和模型架构出发,量化推导 LLM 推理与训练系统设计。含全书正文、PDF、配套计算工具与实验

    GitHub · Python · 8,154 stars · 666 forks · +93 stars in 24 hours · Trending score 3.93

  3. 3
    RunanywhereAI/trnrt

    The open LLM inference engine for AWS Trainium: NKI megakernels, 1M-token context, OpenAI-compatible API. By RunAnywhere.

    GitHub · Python · 1,639 stars · 24 forks · +31 stars in 24 hours · Trending score 3.16

  4. 4
    vllm-project/vllm-ascend

    Community maintained hardware plugin for vLLM on Huawei Ascend

    GitHub · Python · 3,124 stars · 2,788 forks · +24 stars in 24 hours · Trending score 2.96

  5. 5
    ray-project/ray

    Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

    GitHub · Python · 44,347 stars · 8,298 forks · +28 stars in 24 hours · Trending score 2.93

  6. 6
    peonist-ai/halogen-flash-server

    Local inference that behaves like a frontier API: Qwen3.8-Flash-Next on AMD Strix Halo plus small NPU models, over the OpenAI and Anthropic APIs on one port.

    GitHub · Shell · 1,403 stars · 101 forks · +17 stars in 24 hours · Trending score 2.66

  7. 7
    NVIDIA/TensorRT-LLM

    TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.

    GitHub · Python · 15,105 stars · 2,944 forks · +21 stars in 24 hours · Trending score 2.65

  8. 8
    SemiAnalysisAI/InferenceX

    Open Source AI Accelerator Research Platform Standard / 开源推理研究平台

    GitHub · Python · 2,069 stars · 382 forks · +15 stars in 24 hours · Trending score 2.47

  9. 9
    gpustack/gpustack

    A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

    GitHub · Python · 6,086 stars · 746 forks · +11 stars in 24 hours · Trending score 2.42

  10. 10
    OmniJev/Flide

    🔥🔥 The extremely fast inference engine for decision model

    GitHub · Python · 546 stars · 37 forks · +9 stars in 24 hours · Trending score 2.36

  11. 11
    skypilot-org/skypilot

    The AI Compute Platform for frontier teams. SkyPilot turns fragmented AI compute into one AI supercomputer, so frontier AI teams build custom intelligence faster.

    GitHub · Python · 10,862 stars · 1,360 forks · +6 stars in 24 hours · Trending score 2.09

  12. 12
    ekzhang/resin

    🧱 Experimental library for building inference engines

    GitHub · Python · 440 stars · 41 forks · +4 stars in 24 hours · Trending score 2.04

  13. 13
    jaedync/crucible-strix-halo

    Run Qwen3.8-Flash-Next on one AMD Strix Halo (Ryzen AI Max+ 395, gfx1151). EXL3 4.05 and 3.05 bpw packs, own ROCm/HIP kernels, OpenAI-compatible server, 4 sessions of 262K context, setup guide included.

    GitHub · Python · 35 stars · 7 forks · +2 stars in 24 hours · Trending score 1.45

  14. 14
    ARahim3/kaggle-tpu-lab

    Frontier-class open models on a free Kaggle TPU v5e-8: GLM-5.3-Flash 320B MoE (~165 tok/s, our own JAX engine) and Qwen3.8-27B bf16 (~130 tok/s), 262k context, prefix caching. Works with Claude Code, Codex, opencode and pi.

    GitHub · Python · 813 stars · 110 forks · +2 stars in 24 hours · Trending score 1.32

  15. 15
    bentoml/BentoML

    The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

    GitHub · Python · 8,941 stars · 1,065 forks · 0 stars in 24 hours · Trending score 1.20

  16. 16
    aws-samples/sample-genai-on-eks-starter-kit

    A comprehensive toolkit for deploying production-ready Generative AI infrastructure on Amazon EKS. Includes pre-configured components for: 🚀 AI Gateway (LiteLLM) 🤖 LLM Serving (vLLM, SGLang, Ollama) 📊 Vector Databases, 🔍 Embedding Models (TEI) 📈 Observability (Langfuse, Phoenix) etc. Fast-track your GenAI deployment with Kubernetes

    GitHub · JavaScript · 102 stars · 64 forks · +3 stars in 24 hours · Trending score 1.19

  17. 17
    helixml/helix

    ♾️ Private Agent Fleet with Spec Coding. Each agent gets their own GPU-accelerated desktop. Run Claude, Codex, Gemini and open models on a full private AI Stack ♾️

    GitHub · Go · 833 stars · 90 forks · +2 stars in 24 hours · Trending score 1.10

  18. 18
    alibaba/rtp-llm

    RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

    GitHub · Python · 1,403 stars · 307 forks · +1 stars in 24 hours · Trending score 1.09

  19. 19
    Blackwellboy/model-serving-minefield

    Community registry of LLM serving-path traps that produce confidently wrong measurements: templates, tool parsers, reasoning fields, quant kernel paths, CUDA toolchains, KV allocation, eval harnesses, versioning. Symptom-first, with the check that catches each.

    GitHub · Python · 177 stars · 19 forks · 0 stars in 24 hours · Trending score 1.02

  20. 20
    OrbitInfra/OrbitKV

    OrbitKV is a High Performance KV Cache System for LLM Inference.

    GitHub · Rust · 284 stars · 19 forks · +1 stars in 24 hours · Trending score 0.82

  21. 21
    anthony-chaudhary/fak

    Agentic Runtime

    GitHub · Go · 50 stars · 16 forks · +1 stars in 24 hours · Trending score 0.78

  22. 22
    basecompute/superfluid

    Superfluid is an open-source server that runs LLMs on your own hardware. It serves a model from one machine, or from several machines that act as one server. It supports llama.cpp, MLX, and BaseRT, and exposes OpenAI-, Anthropic-, and Ollama-compatible API endpoints.

    GitHub · Rust · 31 stars · 4 forks · 0 stars in 24 hours · Trending score 0.64

  23. 23
    leonxger/deepseek-colab

    Uncensored DeepSeek-V4.1-Flash IQ2_XS on Google Colab

    GitHub · Python · 60 stars · 5 forks · +1 stars in 24 hours · Trending score 0.64

  24. 24
    avifenesh/memra

    Memra is a Rust + CUDA LLM inference engine with hand-written kernels, NVFP4 and mixed-precision execution, and correctness gates scoped to each model and GPU

    GitHub · Rust · 7 stars · 1 forks · +1 stars in 24 hours · Trending score 0.63