interpretability repositories

Trending repositories tagged interpretability, ranked by trending score. Updated daily.

  1. 1
    pyc-team/pytorch_concepts

    PyC (Pytorch Concepts) is a PyTorch-based library for designing concept-based interpretable deep learning models.

    GitHub · Python · 342 stars · 33 forks · +183 stars in 24 hours · Trending score 3.93

  2. 2
    lyramakesmusic/neuron-explorer

    Steering-first neuron explorer for Qwen3.5-4B: push every MLP neuron, keep every rollout, browse the results

    GitHub · Python · 2,041 stars · 254 forks · +17 stars in 24 hours · Trending score 2.47

  3. 3
    hijohnnylin/neuronpedia

    open source interpretability platform 🧠

    GitHub · TypeScript · 1,225 stars · 171 forks · +15 stars in 24 hours · Trending score 2.15

  4. 4
    MirroS-Lab/S-Space

    S-Space: Exploring Spatial Workspace in Multimodal Models

    GitHub · Python · 185 stars · 5 forks · +16 stars in 24 hours · Trending score 1.95

  5. 5
    fulvian/engraft-ngram

    ENGRAFT: write facts into an LLM's n-gram (Engram) memory table as a removable overlay: 100 facts in a 9 MB file, no weights touched, verified on llama.cpp GGUF (Qwen3.8-Flash-Next).

    GitHub · Python · 62 stars · 5 forks · +11 stars in 24 hours · Trending score 1.79

  6. 6
    google-deepmind/detectorscope

    Build AI text detectors whose decisions are mechanistically grounded and human-auditable (EMNLP 2026)

    GitHub · Python · 496 stars · 44 forks · +4 stars in 24 hours · Trending score 1.53

  7. 7
    stdlogits/neuron-lens

    A catalog of every MLP neuron and attention head of Qwen3-8B and Gemma 4 12B, read from the weights through two Jacobian lenses (R and S), from activations, and from their exact token drivers

    GitHub · Python · 60 stars · 5 forks · +5 stars in 24 hours · Trending score 1.44

  8. 8
    evalvitals/evalrx

    Autonomous failure investigation, root-cause analysis, and validated interventions for LLMs, VLMs, and agents.

    GitHub · Python · 18 stars · 2 forks · +6 stars in 24 hours · Trending score 1.40

  9. 9
    whp199/theodolite

    An atlas of all 1,290,240 neurons in Gemma-4-31B-it, a census of what they do when pushed, and the instrument to read them with.

    GitHub · Python · 39 stars · 5 forks · +2 stars in 24 hours · Trending score 1.26

  10. 10
    di37/animated-machine-learning-algorithm

    Watch scikit-learn models learn and predict: live training and inference animations, narrated lessons and Manim videos for California Housing (regression) and Titanic (classification)

    GitHub · Python · 28 stars · 0 forks · +9 stars in 24 hours · Trending score 1.15

  11. 11
    divagr18/qwenputer

    A computer, compiled into the weights of a language model. An exact CPU inside Qwen3.5-4B, with Gated DeltaNet state as RAM.

    GitHub · Python · 830 stars · 113 forks · 0 stars in 24 hours · Trending score 1.08

  12. 12
    microsoft/responsible-ai-toolbox

    Responsible AI Toolbox is a suite of tools providing model and data exploration and assessment user interfaces and libraries that enable a better understanding of AI systems. These interfaces and libraries empower developers and stakeholders of AI systems to develop and monitor AI more responsibly, and take better data-driven actions.

    GitHub · TypeScript · 1,858 stars · 511 forks · 0 stars in 24 hours · Trending score 1.03

  13. 13
    fswolf/luna-local-agent

    A fully local AI agent built from several models working as one: voice, vision, memory, approved tools, and a live view of her reasoning and token-by-token confidence. Runs on LM Studio or llama.cpp. More information at https://luna.imbored.me

    GitHub · Python · 17 stars · 1 forks · +5 stars in 24 hours · Trending score 0.94

  14. 14
    shap/shap

    A game theoretic approach to explain the output of any machine learning model.

    GitHub · Jupyter Notebook · 25,862 stars · 3,796 forks · 0 stars in 24 hours · Trending score 0.94

  15. 15
    manuelgscom/inside-llm

    See inside any GGUF: where two quantizations part ways, token by token, and what each sampler cuts. Live in your browser, powered by llama.cpp (CUDA or Vulkan).

    GitHub · TypeScript · 110 stars · 11 forks · +4 stars in 24 hours · Trending score 0.80

  16. 16
    eomcaleb/weight-space-bench

    Benchmark: can an LLM make a language model better at math by editing its weights directly? (Qwen3.5-0.8B, MATH-500/GSM8K, via OpenRouter)

    GitHub · Python · 18 stars · 2 forks · +1 stars in 24 hours · Trending score 0.73

  17. 17
    astro-weights/astro-weights

    Small course corrections for open LLMs. Agent-driven weight edits. Democratizing AI for all.

    GitHub · Python · 8 stars · 3 forks · +2 stars in 24 hours · Trending score 0.61

  18. 18
    stdlogits/neuron-reflection

    Swap, fuse, remove or split a concept in Qwen3-8B's weights, without training, by mirroring the neurons and attention heads that carry it

    GitHub · Python · 17 stars · 2 forks · +2 stars in 24 hours · Trending score 0.59

  19. 19
    chessformer-lens/chessformer_lens

    A toolkit+visualizer that is designed for mech interp researchers working with chess models that use transformers tokenized by square.

    GitHub · Python · 10 stars · 0 forks · +1 stars in 24 hours · Trending score 0.55

  20. 20
    OpenBMB/ParamMute

    [NeurIPS 2025] ParamMute (a.k.a. PIP-KAG): suppressing knowledge-critical FFNs for faithful retrieval-augmented generation (RAG knowledge conflicts)

    GitHub · Python · 64 stars · 5 forks · +1 stars in 24 hours · Trending score 0.54

  21. 21
    conceptcountry/weight-space-programming

    Gradient-free weight editing reduces overthinking in Qwen3.8-27B — analysis code and evaluation artifacts

    GitHub · Typst · 31 stars · 3 forks · 0 stars in 24 hours · Trending score 0.50

  22. 22
    Julia-XAI/ExplainableAI.jl

    Explainable AI in Julia.

    GitHub · Julia · 119 stars · 3 forks · +1 stars in 24 hours · Trending score 0.45

  23. 23
    johnmarktaylor91/torchlens

    Inspect, visualize and intervene on any operation in any neural network. Capture every activation and gradient, the full computational graph and 800+ metadata fields without changing model code. Supports PyTorch, JAX, TensorFlow, MLX, PaddlePaddle and tinygrad. Validated on 6400+ models. Built for researchers and their AI agents.

    GitHub · Python · 714 stars · 35 forks · 0 stars in 24 hours · Trending score 0.43

  24. 24
    EleutherAI/bergson

    Mapping out the "memory" of neural nets with data attribution

    GitHub · Python · 110 stars · 35 forks · 0 stars in 24 hours · Trending score 0.42