model-serving repositories

Trending repositories tagged model-serving, ranked by trending score. Updated daily.

  1. 1
    vllm-project/vllm

    A high-throughput and memory-efficient inference and serving engine for LLMs

    GitHub · Python · 97,255 stars · 25,073 forks · +109 stars in 24 hours · Trending score 4.26

  2. 2
    vllm-project/vllm-omni

    A framework for efficient model inference with omni-modality models

    GitHub · Python · 7,856 stars · 2,162 forks · +36 stars in 24 hours · Trending score 3.29

  3. 3
    vllm-project/vllm-ascend

    Community maintained hardware plugin for vLLM on Huawei Ascend

    GitHub · Python · 3,124 stars · 2,788 forks · +24 stars in 24 hours · Trending score 2.96

  4. 4
    sgl-project/sglang-omni

    High Performance Serving Runtime For Interactive Super Intelligence System

    GitHub · Python · 1,827 stars · 762 forks · +15 stars in 24 hours · Trending score 2.83

  5. 5
    aryamandarda/tamserve

    Live, multi-camera object tracking with SAM3, SAM 2 and EdgeTAM, on one GPU or many. One model per GPU serves every camera: faster than the original code, about 4x less GPU memory, the same accuracy.

    GitHub · Python · 255 stars · 14 forks · +163 stars in 24 hours · Trending score 2.55

  6. 6
    kserve/kserve

    Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

    GitHub · Go · 6,251 stars · 1,802 forks · +5 stars in 24 hours · Trending score 2.07

  7. 7
    beclab/Olares

    Open-Source Personal Cloud OS for Always-On Agents

    GitHub · Go · 5,368 stars · 340 forks · +6 stars in 24 hours · Trending score 1.67

  8. 8
    ModelTC/LightLLM

    LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.

    GitHub · Python · 4,367 stars · 389 forks · +5 stars in 24 hours · Trending score 1.58

  9. 9
    ollaya-dev/ollaya

    Jev-compatible API server for open decision models. Ollama for decision models: pull and serve 29 model families (Laya, pplx-decider, Clef, Winnow, coba...) locally or self-hosted behind /v1/systemone. The TypeSafe SDK works unchanged: set its base URL and pick an open model. CLI, desktop app, Docker, MCP server, public bench.

    GitHub · Rust · 1,473 stars · 93 forks · +3 stars in 24 hours · Trending score 1.44

  10. 10
    kitops-ml/kitops

    An open source DevOps tool from the CNCF for packaging and versioning AI/ML models, datasets, code, and configuration into an OCI Artifact.

    GitHub · Go · 1,451 stars · 192 forks · +3 stars in 24 hours · Trending score 1.42

  11. 11
    bentoml/BentoML

    The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

    GitHub · Python · 8,941 stars · 1,065 forks · 0 stars in 24 hours · Trending score 1.20

  12. 12
    ome-projects/ome

    Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton

    GitHub · Go · 528 stars · 107 forks · +3 stars in 24 hours · Trending score 1.10

  13. 13
    alibaba/rtp-llm

    RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

    GitHub · Python · 1,403 stars · 307 forks · +1 stars in 24 hours · Trending score 1.09

  14. 14
    openvinotoolkit/model_server

    A scalable inference server for models optimized with OpenVINO™

    GitHub · C++ · 960 stars · 297 forks · −2 stars in 24 hours · Trending score 0.61

  15. 15
    basetenlabs/truss

    The simplest way to serve AI/ML models in production

    GitHub · Python · 1,235 stars · 143 forks · 0 stars in 24 hours · Trending score 0.58

  16. 16
    pegainfer-project/pegainfer

    Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

    GitHub · Rust · 745 stars · 121 forks · 0 stars in 24 hours · Trending score 0.57

  17. 17
    canonical/inference-snaps

    Local inference, optimized for your hardware

    GitHub · Go · 55 stars · 11 forks · 0 stars in 24 hours · Trending score 0.47

  18. 18
    intel/toolkit-manager

    Production-ready platform for deploying OpenAI-compatible LLM inference on Kubernetes. A one-command installer for enterprise generative AI on Intel® Xeon®, with vLLM and other serving engines, secure Keycloak authentication, and built-in observability - from bare metal to a scalable AI endpoint.

    GitHub · Python · 13 stars · 23 forks · 0 stars in 24 hours · Trending score 0.46

  19. 19
    gpustack/gpustack-operator

    Manage heterogeneous accelerators, vGPU slicing, shared KV cache, model delivery, and inference workloads.

    GitHub · Go · 6 stars · 9 forks · 0 stars in 24 hours · Trending score 0.39

  20. 20
    maheshvaikri-code/snapllm

    🔥 Alternative to Ollama — multi-model serving with sub-ms model switching · CPU-only 20B inference for Edge AI · llama.cpp + stablediffusion.cpp

    GitHub · C++ · 44 stars · 4 forks · 0 stars in 24 hours · Trending score 0.32

  21. 21
    batchgen-project/batchgen

    High-Throughput Batch Inference

    GitHub · C++ · 22 stars · 1 forks · 0 stars in 24 hours · Trending score 0.32

  22. 22
    hertz-ai/HARTOS

    An AI-native OS. Models run on your own hardware, nodes federate peer-to-peer with no broker, and the API is OpenAI-compatible. Boots, has its own Wayland compositor, and runs on 8GB. Apache 2.0.

    GitHub · Python · 52 stars · 7 forks · 0 stars in 24 hours · Trending score 0.24

  23. 23
    Lincoln-cn/JAiRouter

    Self-hosted LLM gateway for Java and Spring Boot teams: one OpenAI-compatible endpoint in front of Ollama, vLLM, GPUStack and cloud models, with RBAC, rate limiting, circuit breaking, audit and a visual console.

    GitHub · Java · 27 stars · 11 forks · 0 stars in 24 hours · Trending score 0.24

  24. 24
    intel/inference-toolkit

    Enterprise-grade, high-performance LLM serving on Kubernetes. OpenAI-compatible inference powered by vLLM and OpenVINO™ Model Server behind a unified AI gateway, with NUMA-aware CPU optimization for Intel® Xeon® - production model serving, routing, and security built in.

    GitHub · Python · 8 stars · 15 forks · 0 stars in 24 hours · Trending score 0.19