serving repositories

Trending repositories tagged serving, ranked by trending score. Updated daily.

  1. 1
    skyzh/tiny-llm

    learn LLM inference system on Apple Silicon for systems engineers: build a tiny vLLM + Qwen

    GitHub · Python · 5,802 stars · 538 forks · +55 stars in 24 hours · Trending score 3.24

  2. 2
    Lightning-AI/LitServe

    A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.

    GitHub · Python · 4,016 stars · 334 forks · +37 stars in 24 hours · Trending score 2.74

  3. 3
    ray-project/ray

    Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

    GitHub · Python · 44,352 stars · 8,296 forks · +12 stars in 24 hours · Trending score 2.49

  4. 4
    openvinotoolkit/model_server

    A scalable inference server for models optimized with OpenVINO™

    GitHub · C++ · 962 stars · 297 forks · +2 stars in 24 hours · Trending score 1.14

  5. 5
    volcano-sh/volcano

    A Cloud Native Batch System (Project under CNCF)

    GitHub · Go · 6,098 stars · 1,640 forks · 0 stars in 24 hours · Trending score 0.97

  6. 6
    pixeltable/pixeltable

    The backend agents build with - Multimodal database, orchestration, and serving in one file

    GitHub · Python · 1,664 stars · 245 forks · +1 stars in 24 hours · Trending score 0.90

  7. 7
    dingodb/dingo

    A multi-modal vector database that supports upserts and vector queries using unified SQL (MySQL-Compatible) on structured and unstructured data, while meeting the requirements of high concurrency and ultra-low latency.

    GitHub · Java · 1,706 stars · 263 forks · +3 stars in 24 hours · Trending score 0.90

  8. 8
    deepjavalibrary/djl-serving

    A universal scalable machine learning model deployment solution

    GitHub · Java · 258 stars · 100 forks · 0 stars in 24 hours · Trending score 0.22

  9. 9
    polyaxon/haupt

    Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon

    GitHub · Python · 451 stars · 205 forks · −1 stars in 24 hours · Trending score 0.18

  10. 10
    tensorflow/serving

    A flexible, high-performance serving system for machine learning models

    GitHub · C++ · 6,362 stars · 2,201 forks · 0 stars in 24 hours · Trending score 0.08

  11. 11
    clearml/clearml-serving

    ClearML - Model-Serving Orchestration and Repository Solution

    GitHub · Python · 170 stars · 54 forks · 0 stars in 24 hours · Trending score 0.07

  12. 12
    AI-Hypercomputer/gpu-recipes

    Recipes for reproducing training and serving benchmarks for large machine learning models using GPUs on Google Cloud.

    GitHub · Python · 148 stars · 94 forks · 0 stars in 24 hours · Trending score 0.02

  13. 13
    sgyaqing/verdict

    A high-performance serving engine for System One decision models.

    GitHub · Rust · 13 stars · 0 forks · 0 stars in 24 hours · Trending score 0.02

  14. 14
    PaddlePaddle/FastDeploy

    High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle

    GitHub · Python · 3,722 stars · 756 forks · 0 stars in 24 hours · Trending score 0.00