model-serving repositories
Trending repositories tagged model-serving, ranked by trending score. Updated daily.
-
1
vllm-project/vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
GitHub · Python · 97,255 stars · 25,073 forks · +109 stars in 24 hours · Trending score 4.26
-
2
vllm-project/vllm-omni
A framework for efficient model inference with omni-modality models
GitHub · Python · 7,856 stars · 2,162 forks · +36 stars in 24 hours · Trending score 3.29
-
3
vllm-project/vllm-ascend
Community maintained hardware plugin for vLLM on Huawei Ascend
GitHub · Python · 3,124 stars · 2,788 forks · +24 stars in 24 hours · Trending score 2.96
-
4
sgl-project/sglang-omni
High Performance Serving Runtime For Interactive Super Intelligence System
GitHub · Python · 1,827 stars · 762 forks · +15 stars in 24 hours · Trending score 2.83
-
5
aryamandarda/tamserve
Live, multi-camera object tracking with SAM3, SAM 2 and EdgeTAM, on one GPU or many. One model per GPU serves every camera: faster than the original code, about 4x less GPU memory, the same accuracy.
GitHub · Python · 255 stars · 14 forks · +163 stars in 24 hours · Trending score 2.55
-
6
kserve/kserve
Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes
GitHub · Go · 6,251 stars · 1,802 forks · +5 stars in 24 hours · Trending score 2.07
-
7
beclab/Olares
Open-Source Personal Cloud OS for Always-On Agents
GitHub · Go · 5,368 stars · 340 forks · +6 stars in 24 hours · Trending score 1.67
-
8
ModelTC/LightLLM
LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.
GitHub · Python · 4,367 stars · 389 forks · +5 stars in 24 hours · Trending score 1.58
-
9
ollaya-dev/ollaya
Jev-compatible API server for open decision models. Ollama for decision models: pull and serve 29 model families (Laya, pplx-decider, Clef, Winnow, coba...) locally or self-hosted behind /v1/systemone. The TypeSafe SDK works unchanged: set its base URL and pick an open model. CLI, desktop app, Docker, MCP server, public bench.
GitHub · Rust · 1,473 stars · 93 forks · +3 stars in 24 hours · Trending score 1.44
-
10
kitops-ml/kitops
An open source DevOps tool from the CNCF for packaging and versioning AI/ML models, datasets, code, and configuration into an OCI Artifact.
GitHub · Go · 1,451 stars · 192 forks · +3 stars in 24 hours · Trending score 1.42
-
11
bentoml/BentoML
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
GitHub · Python · 8,941 stars · 1,065 forks · 0 stars in 24 hours · Trending score 1.20
-
12
ome-projects/ome
Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton
GitHub · Go · 528 stars · 107 forks · +3 stars in 24 hours · Trending score 1.10
-
13
alibaba/rtp-llm
RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.
GitHub · Python · 1,403 stars · 307 forks · +1 stars in 24 hours · Trending score 1.09
-
14
openvinotoolkit/model_server
A scalable inference server for models optimized with OpenVINO™
GitHub · C++ · 960 stars · 297 forks · −2 stars in 24 hours · Trending score 0.61
-
15
basetenlabs/truss
The simplest way to serve AI/ML models in production
GitHub · Python · 1,235 stars · 143 forks · 0 stars in 24 hours · Trending score 0.58
-
16
pegainfer-project/pegainfer
Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2
GitHub · Rust · 745 stars · 121 forks · 0 stars in 24 hours · Trending score 0.57
-
17
canonical/inference-snaps
Local inference, optimized for your hardware
GitHub · Go · 55 stars · 11 forks · 0 stars in 24 hours · Trending score 0.47
-
18
intel/toolkit-manager
Production-ready platform for deploying OpenAI-compatible LLM inference on Kubernetes. A one-command installer for enterprise generative AI on Intel® Xeon®, with vLLM and other serving engines, secure Keycloak authentication, and built-in observability - from bare metal to a scalable AI endpoint.
GitHub · Python · 13 stars · 23 forks · 0 stars in 24 hours · Trending score 0.46
-
19
gpustack/gpustack-operator
Manage heterogeneous accelerators, vGPU slicing, shared KV cache, model delivery, and inference workloads.
GitHub · Go · 6 stars · 9 forks · 0 stars in 24 hours · Trending score 0.39
-
20
maheshvaikri-code/snapllm
🔥 Alternative to Ollama — multi-model serving with sub-ms model switching · CPU-only 20B inference for Edge AI · llama.cpp + stablediffusion.cpp
GitHub · C++ · 44 stars · 4 forks · 0 stars in 24 hours · Trending score 0.32
-
21
batchgen-project/batchgen
High-Throughput Batch Inference
GitHub · C++ · 22 stars · 1 forks · 0 stars in 24 hours · Trending score 0.32
-
22
hertz-ai/HARTOS
An AI-native OS. Models run on your own hardware, nodes federate peer-to-peer with no broker, and the API is OpenAI-compatible. Boots, has its own Wayland compositor, and runs on 8GB. Apache 2.0.
GitHub · Python · 52 stars · 7 forks · 0 stars in 24 hours · Trending score 0.24
-
23
Lincoln-cn/JAiRouter
Self-hosted LLM gateway for Java and Spring Boot teams: one OpenAI-compatible endpoint in front of Ollama, vLLM, GPUStack and cloud models, with RBAC, rate limiting, circuit breaking, audit and a visual console.
GitHub · Java · 27 stars · 11 forks · 0 stars in 24 hours · Trending score 0.24
-
24
intel/inference-toolkit
Enterprise-grade, high-performance LLM serving on Kubernetes. OpenAI-compatible inference powered by vLLM and OpenVINO™ Model Server behind a unified AI gateway, with NUMA-aware CPU optimization for Intel® Xeon® - production model serving, routing, and security built in.
GitHub · Python · 8 stars · 15 forks · 0 stars in 24 hours · Trending score 0.19