local-inference repositories
Trending repositories tagged local-inference, ranked by trending score. Updated daily.
-
1
Phocinae/Phocinae-Largha-150M-v1
斑海豹 — Phocinae-Largha-150M-v1: a 144.3M bilingual decision model. Tiny model. Big decisions. 小海豹,大决断。
GitHub · Python · 800 stars · 1 forks · +228 stars in 24 hours · Trending score 3.56
-
2
mozilla-ai/llamafile
Distribute and run LLMs with a single file.
GitHub · C++ · 26,684 stars · 1,684 forks · +113 stars in 24 hours · Trending score 3.42
-
3
huawei-csl/MACaQ
SOTA model-agnostic quantization for GGUF, local inference and beyond. Quantize and run text, image, video and audio models with llama.cpp, vLLM, Diffusers and PyTorch.
GitHub · C++ · 385 stars · 22 forks · +8 stars in 24 hours · Trending score 2.39
-
4
alesha-pro/tools
Tools, ComfyUI workflows and benchmark configs from a 4x RTX 3090 local-inference rig
GitHub · HTML · 702 stars · 82 forks · +15 stars in 24 hours · Trending score 2.05
-
5
Magnvs7/Promethevs
PROMETHEVS: run the full DeepSeek-V4.1-Flash-1017 (510 GB) on a gaming PC with 64 GB RAM and a 12 GB GPU. PROMETHEVS engine (compress-then-correct), OpenAI-compatible local server, report and logs.
GitHub · 193 stars · 13 forks · +2 stars in 24 hours · Trending score 1.30
-
6
san-digital/tokspur
Private chat and code on your Mac. Run a 177B model on a 128 GB M5 Max: local Qwen3.8 Flash-Next inference with a 65,536-token default or optional 131,072-token context, Tokspur's own Metal engine with native MTP or DFlash2 drafting, and an OpenAI-compatible API. Apache 2.0.
GitHub · C++ · 29 stars · 3 forks · +3 stars in 24 hours · Trending score 1.18
-
7
gianlucamazza/xllama
Local LLM inference on Xbox Series S and Series X in Developer Mode: llama.cpp (GGUF) + ONNX Runtime GenAI, an OpenAI-compatible LAN endpoint (off by default), SD-Turbo image generation and experimental on-device fine-tuning. Measured on Series S. Independent project, not affiliated with Microsoft.
GitHub · C++ · 93 stars · 18 forks · +1 stars in 24 hours · Trending score 1.14
-
8
Anionex/h3-image-mlx
Faster local MiniMax H3 Image generation and editing on Apple Silicon with MLX, with ultra-low ram
GitHub · Python · 80 stars · 11 forks · 0 stars in 24 hours · Trending score 1.00
-
9
bladedevoff/stuntd
Local proxy that learns your app's typed LLM decisions and answers them with a Laya head. Jev and OpenAI compatible.
GitHub · Python · 76 stars · 5 forks · +3 stars in 24 hours · Trending score 0.90
-
10
raspoli/mlx-serve
Local inference server for Apple Silicon — hot-swaps MLX models (LLM, vision, embeddings, TTS, STT) via OpenAI API
GitHub · Python · 23 stars · 4 forks · +2 stars in 24 hours · Trending score 0.82
-
11
ahwurm/localharness
An open-source, coding agent harness designed specifically for local LLMs. Designed as everything a user needs to jumpstart their own harness (detailed event steams, logs, and performance data) or quickly replace closed source one (featuring out of the box plugins for web search, image gen, memory, auto research and more in dev).
GitHub · Python · 74 stars · 7 forks · 0 stars in 24 hours · Trending score 0.80
-
12
ollaya-dev/ollaya
Jev-compatible API server for open decision models. Ollama for decision models: pull and serve 29 model families (Laya, pplx-decider, Clef, Winnow, coba...) locally or self-hosted behind /v1/systemone. The TypeSafe SDK works unchanged: set its base URL and pick an open model. CLI, desktop app, Docker, MCP server, public bench.
GitHub · Rust · 1,473 stars · 93 forks · −1 stars in 24 hours · Trending score 0.72
-
13
sahibzada-allahyar/gliner2-ultrafast
Real browser automation with local GLiNER2 inference and open weights. Work in progress. Contributions welcome.
GitHub · Python · 81 stars · 9 forks · +2 stars in 24 hours · Trending score 0.72
-
14
notolog/notolog-editor
Notolog Markdown Editor
GitHub · Python · 31 stars · 10 forks · +1 stars in 24 hours · Trending score 0.66
-
15
NeuroBrix/neurobrix
Run real AI models on the GPU you already own: language, speech, image and video models from one engine and one container format, with or without PyTorch. NVIDIA GPUs, Mac GPUs through Metal, CPU; other GPUs experimental. Apache 2.0.
GitHub · Python · 67 stars · 0 forks · 0 stars in 24 hours · Trending score 0.56
-
16
ChukarAI/Vibe_Assembly
Vibe Assembly v2.5.1 - zero-dollar, infinite-context vibe coding architecture. Pure-Rust multi-agent harness with hash-anchored patches, Vibe Router + Vibe Local zero-cost inference, reasoning-effort control, CLI/TUI, MCP server, and a GPUI Liquid Glass desktop.
GitHub · Rust · 10 stars · 3 forks · +1 stars in 24 hours · Trending score 0.55
-
17
siamcafe/bomllm
Self-hosted Thai AI infrastructure — Mac Mini M4 Pro running TURBO-Fable 27B at 18.5 tok/s for $0.54/day. Fleet of 9 machines, GOD CONFIG v7.0, 40 IRON safety rules. Production-grade Open WebUI + LiteLLM + Ollama MLX stack.
GitHub · Python · 48 stars · 9 forks · 0 stars in 24 hours · Trending score 0.50
-
18
lna-lab/flash-next-8gb
Run a 177B model (Qwen3.8-Flash-Next) on an 8 GB laptop GPU. Measured: 6.6 GiB VRAM, 47.8 GiB RAM, 34-35 tok/s. The n-gram table stays on disk; the experts run on CPU.
GitHub · Python · 36 stars · 10 forks · 0 stars in 24 hours · Trending score 0.49
-
19
ardanlabs/kronk
Go With Your Own Intelligence! Use Go for hardware accelerated local inference with llama.cpp, whisper.cpp, and stable-diffusion.cpp directly integrated into your Go applications. Kronk provides a high-level API and production ready model server.
GitHub · Go · 871 stars · 65 forks · 0 stars in 24 hours · Trending score 0.36
-
20
c1t1zen1/markgitup-research-desk
Autonomous hourly research desk: a local LLM picks a fresh angle from 24 standing AI/policy/markets topic families, SearXNG gathers the evidence, and nothing publishes without 2+ independent sources. Static site, zero dependencies, built with Hermes Agent.
GitHub · Python · 9 stars · 0 forks · 0 stars in 24 hours · Trending score 0.10
-
21
launcher-rs/audio-cpp-rs
audio.cpp(基于 ggml 的本地音频推理引擎)的 Rust 安全绑定与 FFI 封装:TTS / ASR / VAD / 说话人分离 / 音源分离 / 声音克隆 / 音频编解码 / 音乐生成,覆盖 135 个模型族,后端支持 CPU、CUDA、HIP、Vulkan、Metal,提供离线与流式 API 与预编译库自动下载。
GitHub · Rust · 5 stars · 0 forks · 0 stars in 24 hours · Trending score 0.09
-
22
KyaniteLabs/qwen38-27b-strix-halo
Qwen 3.8 27B on a $1,400 Strix Halo: frozen optimal config, 262k needle-exact retrieval, vision fixed, delegation-card certified
GitHub · Python · 14 stars · 0 forks · 0 stars in 24 hours · Trending score 0.06
-
23
hossbit/local-ai-server
Run GGUF language models locally with llama.cpp, llama-swap, CUDA/Vulkan GPU acceleration, automatic model discovery, systemd integration, and an OpenAI-compatible API.
GitHub · Shell · 8 stars · 1 forks · 0 stars in 24 hours · Trending score 0.02
-
24
loomax-labs/InferrLM
InferrLM - On-device AI for iOS & Android
GitHub · TypeScript · 113 stars · 24 forks · 0 stars in 24 hours · Trending score 0.00