interpretability repositories
Trending repositories tagged interpretability, ranked by trending score. Updated daily.
-
1
pyc-team/pytorch_concepts
PyC (Pytorch Concepts) is a PyTorch-based library for designing concept-based interpretable deep learning models.
GitHub · Python · 342 stars · 33 forks · +183 stars in 24 hours · Trending score 3.93
-
2
lyramakesmusic/neuron-explorer
Steering-first neuron explorer for Qwen3.5-4B: push every MLP neuron, keep every rollout, browse the results
GitHub · Python · 2,041 stars · 254 forks · +17 stars in 24 hours · Trending score 2.47
-
3
hijohnnylin/neuronpedia
open source interpretability platform 🧠
GitHub · TypeScript · 1,225 stars · 171 forks · +15 stars in 24 hours · Trending score 2.15
-
4
MirroS-Lab/S-Space
S-Space: Exploring Spatial Workspace in Multimodal Models
GitHub · Python · 185 stars · 5 forks · +16 stars in 24 hours · Trending score 1.95
-
5
fulvian/engraft-ngram
ENGRAFT: write facts into an LLM's n-gram (Engram) memory table as a removable overlay: 100 facts in a 9 MB file, no weights touched, verified on llama.cpp GGUF (Qwen3.8-Flash-Next).
GitHub · Python · 62 stars · 5 forks · +11 stars in 24 hours · Trending score 1.79
-
6
google-deepmind/detectorscope
Build AI text detectors whose decisions are mechanistically grounded and human-auditable (EMNLP 2026)
GitHub · Python · 496 stars · 44 forks · +4 stars in 24 hours · Trending score 1.53
-
7
stdlogits/neuron-lens
A catalog of every MLP neuron and attention head of Qwen3-8B and Gemma 4 12B, read from the weights through two Jacobian lenses (R and S), from activations, and from their exact token drivers
GitHub · Python · 60 stars · 5 forks · +5 stars in 24 hours · Trending score 1.44
-
8
evalvitals/evalrx
Autonomous failure investigation, root-cause analysis, and validated interventions for LLMs, VLMs, and agents.
GitHub · Python · 18 stars · 2 forks · +6 stars in 24 hours · Trending score 1.40
-
9
whp199/theodolite
An atlas of all 1,290,240 neurons in Gemma-4-31B-it, a census of what they do when pushed, and the instrument to read them with.
GitHub · Python · 39 stars · 5 forks · +2 stars in 24 hours · Trending score 1.26
-
10
di37/animated-machine-learning-algorithm
Watch scikit-learn models learn and predict: live training and inference animations, narrated lessons and Manim videos for California Housing (regression) and Titanic (classification)
GitHub · Python · 28 stars · 0 forks · +9 stars in 24 hours · Trending score 1.15
-
11
divagr18/qwenputer
A computer, compiled into the weights of a language model. An exact CPU inside Qwen3.5-4B, with Gated DeltaNet state as RAM.
GitHub · Python · 830 stars · 113 forks · 0 stars in 24 hours · Trending score 1.08
-
12
microsoft/responsible-ai-toolbox
Responsible AI Toolbox is a suite of tools providing model and data exploration and assessment user interfaces and libraries that enable a better understanding of AI systems. These interfaces and libraries empower developers and stakeholders of AI systems to develop and monitor AI more responsibly, and take better data-driven actions.
GitHub · TypeScript · 1,858 stars · 511 forks · 0 stars in 24 hours · Trending score 1.03
-
13
fswolf/luna-local-agent
A fully local AI agent built from several models working as one: voice, vision, memory, approved tools, and a live view of her reasoning and token-by-token confidence. Runs on LM Studio or llama.cpp. More information at https://luna.imbored.me
GitHub · Python · 17 stars · 1 forks · +5 stars in 24 hours · Trending score 0.94
-
14
shap/shap
A game theoretic approach to explain the output of any machine learning model.
GitHub · Jupyter Notebook · 25,862 stars · 3,796 forks · 0 stars in 24 hours · Trending score 0.94
-
15
manuelgscom/inside-llm
See inside any GGUF: where two quantizations part ways, token by token, and what each sampler cuts. Live in your browser, powered by llama.cpp (CUDA or Vulkan).
GitHub · TypeScript · 110 stars · 11 forks · +4 stars in 24 hours · Trending score 0.80
-
16
eomcaleb/weight-space-bench
Benchmark: can an LLM make a language model better at math by editing its weights directly? (Qwen3.5-0.8B, MATH-500/GSM8K, via OpenRouter)
GitHub · Python · 18 stars · 2 forks · +1 stars in 24 hours · Trending score 0.73
-
17
astro-weights/astro-weights
Small course corrections for open LLMs. Agent-driven weight edits. Democratizing AI for all.
GitHub · Python · 8 stars · 3 forks · +2 stars in 24 hours · Trending score 0.61
-
18
stdlogits/neuron-reflection
Swap, fuse, remove or split a concept in Qwen3-8B's weights, without training, by mirroring the neurons and attention heads that carry it
GitHub · Python · 17 stars · 2 forks · +2 stars in 24 hours · Trending score 0.59
-
19
chessformer-lens/chessformer_lens
A toolkit+visualizer that is designed for mech interp researchers working with chess models that use transformers tokenized by square.
GitHub · Python · 10 stars · 0 forks · +1 stars in 24 hours · Trending score 0.55
-
20
OpenBMB/ParamMute
[NeurIPS 2025] ParamMute (a.k.a. PIP-KAG): suppressing knowledge-critical FFNs for faithful retrieval-augmented generation (RAG knowledge conflicts)
GitHub · Python · 64 stars · 5 forks · +1 stars in 24 hours · Trending score 0.54
-
21
conceptcountry/weight-space-programming
Gradient-free weight editing reduces overthinking in Qwen3.8-27B — analysis code and evaluation artifacts
GitHub · Typst · 31 stars · 3 forks · 0 stars in 24 hours · Trending score 0.50
-
22
Julia-XAI/ExplainableAI.jl
Explainable AI in Julia.
GitHub · Julia · 119 stars · 3 forks · +1 stars in 24 hours · Trending score 0.45
-
23
johnmarktaylor91/torchlens
Inspect, visualize and intervene on any operation in any neural network. Capture every activation and gradient, the full computational graph and 800+ metadata fields without changing model code. Supports PyTorch, JAX, TensorFlow, MLX, PaddlePaddle and tinygrad. Validated on 6400+ models. Built for researchers and their AI agents.
GitHub · Python · 714 stars · 35 forks · 0 stars in 24 hours · Trending score 0.43
-
24
EleutherAI/bergson
Mapping out the "memory" of neural nets with data attribution
GitHub · Python · 110 stars · 35 forks · 0 stars in 24 hours · Trending score 0.42