gpu-inference repositories
Trending repositories tagged gpu-inference, ranked by trending score. Updated daily.
-
1
peonist-ai/halogen-flash-server
Local inference that behaves like a frontier API: Qwen3.8-Flash-Next on AMD Strix Halo plus small NPU models, over the OpenAI and Anthropic APIs on one port.
GitHub · Shell · 1,403 stars · 101 forks · +17 stars in 24 hours · Trending score 2.66
-
2
aws-samples/sample-genai-on-eks-starter-kit
A comprehensive toolkit for deploying production-ready Generative AI infrastructure on Amazon EKS. Includes pre-configured components for: 🚀 AI Gateway (LiteLLM) 🤖 LLM Serving (vLLM, SGLang, Ollama) 📊 Vector Databases, 🔍 Embedding Models (TEI) 📈 Observability (Langfuse, Phoenix) etc. Fast-track your GenAI deployment with Kubernetes
GitHub · JavaScript · 102 stars · 64 forks · +3 stars in 24 hours · Trending score 1.19
-
3
shinbunbun/llama-cpp-p100-patches
llama.cpp performance patches developed and measured on Tesla P100 (GP100, sm_60). Most are not Pascal-specific.
GitHub · Nix · 64 stars · 5 forks · 0 stars in 24 hours · Trending score 0.76
-
4
infercrane/infercrane
Open-source infrastructure for the full inference lifecycle: deploy, observe, scale, optimize, and safely release self-hosted models behind one endpoint.
GitHub · Go · 32 stars · 0 forks · −8 stars in 24 hours · Trending score 0.08
-
5
Raytorin/triton-openai-gateway
OpenAI-compatible gateway for NVIDIA Triton and vLLM with tools, multimodal inputs, embeddings, reranking, and observability.
GitHub · Python · 10 stars · 2 forks · 0 stars in 24 hours · Trending score 0.08
-
6
brontoguana/krasis
Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware
GitHub · C++ · 522 stars · 33 forks · 0 stars in 24 hours · Trending score 0.00
-
7
redaER7/qwen3.8-27b-self-hosted
Self-hosted Qwen3.8-27B (FP8) inference with vLLM, KServe and Envoy AI Gateway on RTX 6000 PRo or 2× RTX 4080 Super
GitHub · Shell · 18 stars · 2 forks · 0 stars in 24 hours · Trending score 0.00
-
8
peonist-ai/halogen-server
The fastest way to run Qwen3.8 27B on Strix Halo (gfx1151)
GitHub · Python · 137 stars · 7 forks · 0 stars in 24 hours · Trending score 0.00
-
9
janishar/h3c-studio
Local web UI for h3.c — native Metal video/audio generation with MiniMax-H3 on Apple Silicon, no PyTorch or MLX required.
GitHub · Go · 12 stars · 4 forks · 0 stars in 24 hours · Trending score 0.00
-
10
yeungtuzi/Xiaotu-LLM
XTU = X Transformers Unity — hybrid (CPU-expert + GPU) inference acceleration for vLLM: mainline plugin + CPU MoE engine (AVX512-VNNI/BF16 MXFP4/FP8), long-prefill GPU streaming, SM80 (A100) port.
GitHub · Python · 16 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00