continuous-batching repositories
Trending repositories tagged continuous-batching, ranked by trending score. Updated daily.
-
1
raullenchai/Rapid-MLX
Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm) on the same weights.
GitHub · Python · 4,166 stars · 446 forks · +7 stars in 24 hours · Trending score 2.04
-
2
basecompute/superfluid
Superfluid is an open-source server that runs LLMs on your own hardware. It serves a model from one machine, or from several machines that act as one server. It supports llama.cpp, MLX, and BaseRT, and exposes OpenAI-, Anthropic-, and Ollama-compatible API endpoints.
GitHub · Rust · 31 stars · 4 forks · 0 stars in 24 hours · Trending score 0.64
-
3
Prajwal7-2002/mini-infer
Minimal, educational continuous-batching LLM inference engine with Paged KV-cache, prefix caching, and speculative decoding for Qwen2.5 in pure PyTorch.
GitHub · Python · 5 stars · 0 forks · 0 stars in 24 hours · Trending score 0.05
-
4
waybarrios/vllm-mlx
High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.
GitHub · Python · 1,646 stars · 237 forks · 0 stars in 24 hours · Trending score 0.00
-
5
gty111/gLLM
An Efficient and Versatile Inference Engine for Distributed LLM Serving
GitHub · Python · 74 stars · 8 forks · 0 stars in 24 hours · Trending score 0.00
-
6
harleyszhang/rapid_llm
RapidLLM is an LLM inference framework with continuous batching, tensor and data parallelism, quantized weights, and with pluggable Triton/CUDA kernels.
GitHub · Python · 195 stars · 33 forks · 0 stars in 24 hours · Trending score 0.00
-
7
xarillian/chorus-llm
A local LLM runtime for many-agent applications, built on llama.cpp. One model, shared inference, many agents, embedded. C++/C, Vulkan, Metal, Godot.
GitHub · C++ · 20 stars · 4 forks · 0 stars in 24 hours · Trending score 0.00