inference-server repositories

Trending repositories tagged inference-server, ranked by trending score. Updated daily.

  1. 1
    jundot/omlx

    LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar

    GitHub · Python · 23,761 stars · 2,156 forks · +58 stars in 24 hours · Trending score 3.50

  2. 2
    raullenchai/Rapid-MLX

    Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm) on the same weights.

    GitHub · Python · 4,166 stars · 446 forks · +7 stars in 24 hours · Trending score 2.04

  3. 3
    superlinked/sie

    Open-source inference server and production cluster for all the models your agent needs.

    GitHub · Python · 3,533 stars · 339 forks · +8 stars in 24 hours · Trending score 1.98

  4. 4
    ollaya-dev/ollaya

    Jev-compatible API server for open decision models. Ollama for decision models: pull and serve 29 model families (Laya, pplx-decider, Clef, Winnow, coba...) locally or self-hosted behind /v1/systemone. The TypeSafe SDK works unchanged: set its base URL and pick an open model. CLI, desktop app, Docker, MCP server, public bench.

    GitHub · Rust · 1,473 stars · 93 forks · +3 stars in 24 hours · Trending score 1.44

  5. 5
    roboflow/inference

    Turn any computer or edge device into a command center for your computer vision projects.

    GitHub · Python · 2,498 stars · 342 forks · +1 stars in 24 hours · Trending score 0.90

  6. 6
    basecompute/superfluid

    Superfluid is an open-source server that runs LLMs on your own hardware. It serves a model from one machine, or from several machines that act as one server. It supports llama.cpp, MLX, and BaseRT, and exposes OpenAI-, Anthropic-, and Ollama-compatible API endpoints.

    GitHub · Rust · 31 stars · 4 forks · 0 stars in 24 hours · Trending score 0.64

  7. 7
    basetenlabs/truss

    The simplest way to serve AI/ML models in production

    GitHub · Python · 1,235 stars · 143 forks · 0 stars in 24 hours · Trending score 0.58

  8. 8
    intel/toolkit-manager

    Production-ready platform for deploying OpenAI-compatible LLM inference on Kubernetes. A one-command installer for enterprise generative AI on Intel® Xeon®, with vLLM and other serving engines, secure Keycloak authentication, and built-in observability - from bare metal to a scalable AI endpoint.

    GitHub · Python · 13 stars · 23 forks · 0 stars in 24 hours · Trending score 0.46

  9. 9
    containers/podman-desktop-extension-ai-lab

    Work with LLMs on a local environment using containers

    GitHub · TypeScript · 304 stars · 86 forks · 0 stars in 24 hours · Trending score 0.42

  10. 10
    Michael-A-Kuykendall/shimmy

    ⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.

    GitHub · Rust · 6,014 stars · 589 forks · 0 stars in 24 hours · Trending score 0.20

  11. 11
    chezgoulet/phonon

    Old phones as inference servers for local agentic AI.

    GitHub · Kotlin · 7 stars · 2 forks · 0 stars in 24 hours · Trending score 0.20

  12. 12
    kibae/onnxruntime-server

    ONNX Runtime Server: The ONNX Runtime Server is a server that provides TCP and HTTP/HTTPS REST APIs for ONNX inference.

    GitHub · C++ · 200 stars · 18 forks · 0 stars in 24 hours · Trending score 0.20

  13. 13
    aminry/decisio

    A serving layer for decisions: typed questions in, a probability for every option out, one forward pass each. Three bases: Qwen3.6-35B-A3B, Gemma 4 12B, Gemma 4 31B.

    GitHub · Python · 10 stars · 2 forks · 0 stars in 24 hours · Trending score 0.20

  14. 14
    intel/inference-toolkit

    Enterprise-grade, high-performance LLM serving on Kubernetes. OpenAI-compatible inference powered by vLLM and OpenVINO™ Model Server behind a unified AI gateway, with NUMA-aware CPU optimization for Intel® Xeon® - production model serving, routing, and security built in.

    GitHub · Python · 8 stars · 15 forks · 0 stars in 24 hours · Trending score 0.19

  15. 15
    ai-lab-tech/triton-control

    Web UI for NVIDIA Triton Inference Server on Kubernetes

    GitHub · Python · 10 stars · 7 forks · 0 stars in 24 hours · Trending score 0.10

  16. 16
    modelship-ai/modelship

    Self-hosted, OpenAI-compatible inference for the agentic era: reasoning LLMs, universal tool calling, and the Responses API alongside embeddings, speech, and image models — many models sharing your GPUs, one gateway. Powered by Ray Serve.

    GitHub · Python · 54 stars · 4 forks · 0 stars in 24 hours · Trending score 0.10

  17. 17
    redhat-performance/configiq

    Distributed inference configuration tools

    GitHub · TypeScript · 23 stars · 13 forks · 0 stars in 24 hours · Trending score 0.10

  18. 18
    mxxm-t/llmx

    C++ LLM inference runtime built from scratch, with CPU and Vulkan backends, GGUF support, and multi-GPU serving.

    GitHub · C++ · 5 stars · 0 forks · 0 stars in 24 hours · Trending score 0.10

  19. 19
    AI-native-Systems-Research/scratchy

    Inference serving... from scratch

    GitHub · Rust · 6 stars · 5 forks · 0 stars in 24 hours · Trending score 0.08

  20. 20
    Cedriceuh/freetoken-rdna3

    Fast local LLM inference on AMD Radeon RX 7900 XTX / XT (RDNA3, ROCm): big Mixture-of-Experts models like Qwen3.8-Flash-Next on one or two consumer GPUs. Up to 4 parallel agents with 262k context each, or one 1M-token context, OpenAI/Anthropic API, NVFP4 and EXL3 checkpoints. A tuned build of FreeToken.

    GitHub · Python · 12 stars · 0 forks · 0 stars in 24 hours · Trending score 0.07

  21. 21
    Pushkinist/rMLX

    Native Rust, single-binary MLX inference server for Apple Silicon. OpenAI/Anthropic-compatible LLM serving — text, vision, audio, embeddings — with the widest weight×KV quantization matrix of any MLX server. No Python, no GGUF.

    GitHub · Rust · 18 stars · 7 forks · 0 stars in 24 hours · Trending score 0.03

  22. 22
    zoompilot/jetlink

    Runs openpilot's large driving model on Jetson Orin Nano, macOS, iOS, Android, CUDA laptops with only your Comma, and a USB3 cable

    GitHub · Swift · 50 stars · 36 forks · 0 stars in 24 hours · Trending score 0.01

  23. 23
    waybarrios/vllm-mlx

    High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.

    GitHub · Python · 1,646 stars · 237 forks · 0 stars in 24 hours · Trending score 0.00

  24. 24
    rsasaki0109/vla_ros2

    ROS2-native runtime, benchmark, and adapter hub for Vision-Language-Action models.

    GitHub · Python · 12 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00