inference-server repositories
Trending repositories tagged inference-server, ranked by trending score. Updated daily.
-
1
jundot/omlx
LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar
GitHub · Python · 23,761 stars · 2,156 forks · +58 stars in 24 hours · Trending score 3.50
-
2
raullenchai/Rapid-MLX
Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm) on the same weights.
GitHub · Python · 4,166 stars · 446 forks · +7 stars in 24 hours · Trending score 2.04
-
3
superlinked/sie
Open-source inference server and production cluster for all the models your agent needs.
GitHub · Python · 3,533 stars · 339 forks · +8 stars in 24 hours · Trending score 1.98
-
4
ollaya-dev/ollaya
Jev-compatible API server for open decision models. Ollama for decision models: pull and serve 29 model families (Laya, pplx-decider, Clef, Winnow, coba...) locally or self-hosted behind /v1/systemone. The TypeSafe SDK works unchanged: set its base URL and pick an open model. CLI, desktop app, Docker, MCP server, public bench.
GitHub · Rust · 1,473 stars · 93 forks · +3 stars in 24 hours · Trending score 1.44
-
5
roboflow/inference
Turn any computer or edge device into a command center for your computer vision projects.
GitHub · Python · 2,498 stars · 342 forks · +1 stars in 24 hours · Trending score 0.90
-
6
basecompute/superfluid
Superfluid is an open-source server that runs LLMs on your own hardware. It serves a model from one machine, or from several machines that act as one server. It supports llama.cpp, MLX, and BaseRT, and exposes OpenAI-, Anthropic-, and Ollama-compatible API endpoints.
GitHub · Rust · 31 stars · 4 forks · 0 stars in 24 hours · Trending score 0.64
-
7
basetenlabs/truss
The simplest way to serve AI/ML models in production
GitHub · Python · 1,235 stars · 143 forks · 0 stars in 24 hours · Trending score 0.58
-
8
intel/toolkit-manager
Production-ready platform for deploying OpenAI-compatible LLM inference on Kubernetes. A one-command installer for enterprise generative AI on Intel® Xeon®, with vLLM and other serving engines, secure Keycloak authentication, and built-in observability - from bare metal to a scalable AI endpoint.
GitHub · Python · 13 stars · 23 forks · 0 stars in 24 hours · Trending score 0.46
-
9
containers/podman-desktop-extension-ai-lab
Work with LLMs on a local environment using containers
GitHub · TypeScript · 304 stars · 86 forks · 0 stars in 24 hours · Trending score 0.42
-
10
Michael-A-Kuykendall/shimmy
⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.
GitHub · Rust · 6,014 stars · 589 forks · 0 stars in 24 hours · Trending score 0.20
-
11
chezgoulet/phonon
Old phones as inference servers for local agentic AI.
GitHub · Kotlin · 7 stars · 2 forks · 0 stars in 24 hours · Trending score 0.20
-
12
kibae/onnxruntime-server
ONNX Runtime Server: The ONNX Runtime Server is a server that provides TCP and HTTP/HTTPS REST APIs for ONNX inference.
GitHub · C++ · 200 stars · 18 forks · 0 stars in 24 hours · Trending score 0.20
-
13
aminry/decisio
A serving layer for decisions: typed questions in, a probability for every option out, one forward pass each. Three bases: Qwen3.6-35B-A3B, Gemma 4 12B, Gemma 4 31B.
GitHub · Python · 10 stars · 2 forks · 0 stars in 24 hours · Trending score 0.20
-
14
intel/inference-toolkit
Enterprise-grade, high-performance LLM serving on Kubernetes. OpenAI-compatible inference powered by vLLM and OpenVINO™ Model Server behind a unified AI gateway, with NUMA-aware CPU optimization for Intel® Xeon® - production model serving, routing, and security built in.
GitHub · Python · 8 stars · 15 forks · 0 stars in 24 hours · Trending score 0.19
-
15
ai-lab-tech/triton-control
Web UI for NVIDIA Triton Inference Server on Kubernetes
GitHub · Python · 10 stars · 7 forks · 0 stars in 24 hours · Trending score 0.10
-
16
modelship-ai/modelship
Self-hosted, OpenAI-compatible inference for the agentic era: reasoning LLMs, universal tool calling, and the Responses API alongside embeddings, speech, and image models — many models sharing your GPUs, one gateway. Powered by Ray Serve.
GitHub · Python · 54 stars · 4 forks · 0 stars in 24 hours · Trending score 0.10
-
17
redhat-performance/configiq
Distributed inference configuration tools
GitHub · TypeScript · 23 stars · 13 forks · 0 stars in 24 hours · Trending score 0.10
-
18
mxxm-t/llmx
C++ LLM inference runtime built from scratch, with CPU and Vulkan backends, GGUF support, and multi-GPU serving.
GitHub · C++ · 5 stars · 0 forks · 0 stars in 24 hours · Trending score 0.10
-
19
AI-native-Systems-Research/scratchy
Inference serving... from scratch
GitHub · Rust · 6 stars · 5 forks · 0 stars in 24 hours · Trending score 0.08
-
20
Cedriceuh/freetoken-rdna3
Fast local LLM inference on AMD Radeon RX 7900 XTX / XT (RDNA3, ROCm): big Mixture-of-Experts models like Qwen3.8-Flash-Next on one or two consumer GPUs. Up to 4 parallel agents with 262k context each, or one 1M-token context, OpenAI/Anthropic API, NVFP4 and EXL3 checkpoints. A tuned build of FreeToken.
GitHub · Python · 12 stars · 0 forks · 0 stars in 24 hours · Trending score 0.07
-
21
Pushkinist/rMLX
Native Rust, single-binary MLX inference server for Apple Silicon. OpenAI/Anthropic-compatible LLM serving — text, vision, audio, embeddings — with the widest weight×KV quantization matrix of any MLX server. No Python, no GGUF.
GitHub · Rust · 18 stars · 7 forks · 0 stars in 24 hours · Trending score 0.03
-
22
zoompilot/jetlink
Runs openpilot's large driving model on Jetson Orin Nano, macOS, iOS, Android, CUDA laptops with only your Comma, and a USB3 cable
GitHub · Swift · 50 stars · 36 forks · 0 stars in 24 hours · Trending score 0.01
-
23
waybarrios/vllm-mlx
High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.
GitHub · Python · 1,646 stars · 237 forks · 0 stars in 24 hours · Trending score 0.00
-
24
rsasaki0109/vla_ros2
ROS2-native runtime, benchmark, and adapter hub for Vision-Language-Action models.
GitHub · Python · 12 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00