continuous-batching repositories

Trending repositories tagged continuous-batching, ranked by trending score. Updated daily.

  1. 1
    raullenchai/Rapid-MLX

    Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm) on the same weights.

    GitHub · Python · 4,166 stars · 446 forks · +7 stars in 24 hours · Trending score 2.04

  2. 2
    basecompute/superfluid

    Superfluid is an open-source server that runs LLMs on your own hardware. It serves a model from one machine, or from several machines that act as one server. It supports llama.cpp, MLX, and BaseRT, and exposes OpenAI-, Anthropic-, and Ollama-compatible API endpoints.

    GitHub · Rust · 31 stars · 4 forks · 0 stars in 24 hours · Trending score 0.64

  3. 3
    Prajwal7-2002/mini-infer

    Minimal, educational continuous-batching LLM inference engine with Paged KV-cache, prefix caching, and speculative decoding for Qwen2.5 in pure PyTorch.

    GitHub · Python · 5 stars · 0 forks · 0 stars in 24 hours · Trending score 0.05

  4. 4
    waybarrios/vllm-mlx

    High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.

    GitHub · Python · 1,646 stars · 237 forks · 0 stars in 24 hours · Trending score 0.00

  5. 5
    gty111/gLLM

    An Efficient and Versatile Inference Engine for Distributed LLM Serving

    GitHub · Python · 74 stars · 8 forks · 0 stars in 24 hours · Trending score 0.00

  6. 6
    harleyszhang/rapid_llm

    RapidLLM is an LLM inference framework with continuous batching, tensor and data parallelism, quantized weights, and with pluggable Triton/CUDA kernels.

    GitHub · Python · 195 stars · 33 forks · 0 stars in 24 hours · Trending score 0.00

  7. 7
    xarillian/chorus-llm

    A local LLM runtime for many-agent applications, built on llama.cpp. One model, shared inference, many agents, embedded. C++/C, Vulkan, Metal, Godot.

    GitHub · C++ · 20 stars · 4 forks · 0 stars in 24 hours · Trending score 0.00