General attribution and intervention
Input attribution and internal causal claims use different interventions.
LLM interpretability, editing, and unlearning
Locating a component, assigning a readable label, changing a fact, and proving removal are separate scientific claims with different interventions and tests.
11 maintained papers · literature reviewed through 2 October 2026
Attribution, circuits, sparse features, weight editing, and unlearning answer different questions. Removing hazardous benchmark performance is not proof that information is absent: localization, causal mediation, edit efficacy, specificity, side effects, persistence, relearning, and adaptive attacks require separate evidence.
Prerequisite contract
Transformer architecture
Attention, residual streams, and MLPs define candidate mechanisms.
How the field developed
2017
Input attribution prerequisites
SHAP and Integrated Gradients clarify additive and path-based attribution assumptions before moving inside a language model.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 01.01 A Unified Approach to Interpreting Model Predictions SHAP defines a local explanation as an additive model over binary simplified features and proves that local accuracy, missingness, and consistency select one attribution: the Shapley value of a conditional-expectation game. The base value is the expected model output; feature attributions move that baseline to the prediction for the case being explained. The framework unifies several earlier additive explanation methods while making their hidden choices about missing features explicit. Official implementation 01.02 Axiomatic Attribution for Deep Networks Integrated Gradients attributes a model’s output difference between an input and a baseline by integrating ordinary input gradients along the straight path joining them. Multiplying each average path gradient by the input displacement yields feature credits that satisfy sensitivity, implementation invariance, and completeness. It needs no network modification and uses repeated gradient evaluations, but the answer remains relative to the selected baseline and output. — Work: Vary baselines, token groupings, target logits, and paraphrases for one generation.
Exit evidence: State whether the result is sensitivity, contribution, or causal mediation.
2021–2022
Circuits, causal tracing, and single edits
Transformer Circuits develops residual and attention composition tools; ROME causally traces factual recall and performs a rank-one MLP update.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 02.01 A Mathematical Framework for Transformer CircuitsAuthor-published technical article; not presented as a peer-reviewed conference paper. A Mathematical Framework for Transformer Circuits rewrites attention-only Transformers as sums of end-to-end computational paths through a shared residual stream. Each attention head separates into a QK circuit that decides where to read and an OV circuit that decides what output-direction change to write. Multiplying read/write matrices exposes virtual weights and head composition without treating intermediate query, key, and value vectors as the final explanatory objects. — 02.02 Locating and Editing Factual Associations in GPT ROME first uses causal tracing to locate computations decisive for recalling a subject–relation fact, then edits one middle-layer MLP projection with a rank-one update. The update makes the MLP map the subject’s key representation to a newly optimized value that raises the desired object, while a covariance-weighted constraint minimizes interference with other stored mappings. Official project and code Work: Localize a behavior, intervene on candidate components, then test one edit with counterfactual controls.
Exit evidence: Report efficacy, paraphrase generalization, specificity, and unrelated-task change.
2023–2024
Mass editing and sparse features
MEMIT distributes updates for many facts across layers; sparse autoencoders learn feature dictionaries intended to reduce superposition.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 03.01 Mass-Editing Memory in a Transformer MEMIT scales the associative-memory idea behind ROME from one fact in one layer to thousands of facts distributed across a range of causal middle-layer MLPs. It optimizes a target value for each subject–relation–object edit, divides the required residual change across selected layers, and solves batched covariance-regularized matrix updates that insert many key–value associations simultaneously. Official code 03.02 Sparse Autoencoders Find Highly Interpretable Features in Language Models Cunningham et al. train overcomplete sparse autoencoders on internal Pythia activations to recover feature directions that need not align with individual neurons. A ReLU encoder maps each activation to sparse nonnegative coefficients; tied decoder directions reconstruct the activation. An L1 penalty trades reconstruction fidelity for sparsity, aiming to separate features that were superposed in the model’s native basis. — Work: Stress edit scale, sequential persistence, feature reconstruction, sparsity, and behavioral intervention.
Exit evidence: Measure side effects and coverage rather than counting readable labels.
2024
Hazardous-knowledge evaluation and unlearning
WMDP creates hazardous-knowledge evaluations and tests representation misdirection, coupling an unlearning intervention to retained-utility and attack-aware measurement.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 04.01 The WMDP Benchmark: Measuring and Reducing Malicious Use with Unlearning WMDP supplies 3,668 expert-curated multiple-choice questions in biosecurity, cybersecurity, and chemical security as proxy measurements of hazardous knowledge. The same paper introduces Representation Misdirection for Unlearning (RMU): fine-tune a few early-to-middle layers so hazardous-text activations approach a fixed random direction with large norm, while benign-text activations remain close to the frozen original model. Success requires reduced WMDP-Bio/Cyber accuracy without collapsing general utility. Official benchmark and RMU code · Project website Work: Measure target-domain suppression, broad utility, relearning, and white-box extraction before claiming unlearning.
Exit evidence: State the threat model and show which evidence distinguishes suppression from removal.
2026
Localized unlearning and cross-modal steering
PALU localizes entropy maximization to sensitive prefixes/logits; CoSToM uses causal tracing for activation steering; textual vectors and multimodal-ICL analysis test whether language-derived mechanisms transfer into vision-language descendants.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 05.01 Maximizing Local Entropy Where It Matters: Prefix-Aware Localized LLM UnlearningPeer-reviewed ACL 2026 long paper and Outstanding Paper award recipient. PALU makes LLM unlearning sparse along two axes. At the sequence axis it intervenes only at the first few tokens that initiate each sensitive span, preserves common tokens with a KL constraint, and ignores later redundant sensitive tokens. At the vocabulary axis it maximizes entropy only among the top-K next-token logits. The aim is to disrupt a memorized generation trajectory while perturbing fewer positions and fewer vocabulary dimensions than full-distribution entropy maximization. Author manuscript · Official code 05.02 CoSToM: Causal-oriented Steering for Intrinsic Theory-of-Mind Alignment in Large Language ModelsPeer-reviewed ACL 2026 long paper and Best Theme Paper award recipient. CoSToM turns a mechanistic probe into a training interface for Theory-of-Mind behavior. One copy of an LLM encodes a dialogue, a frozen second copy receives a patched intermediate activation and answers belief–desire–intention questions, and the resulting loss is backpropagated through the patch into shallow LoRA adapters of the encoder. The tuned encoder is then used alone to generate dialogue whose behavior is intended to reflect the inferred mental state. Author manuscript · Official code 05.03 Textual Steering Vectors Can Improve Visual Understanding in Multimodal Large Language ModelsPeer-reviewed ACL 2026 long paper; catalog authors follow the final proceedings record. This paper discovers steering directions using text-only activations in a language-model backbone and applies those directions inside a related multimodal model. Sparse-autoencoder features, class-mean differences, or linear probes define vectors associated with concepts such as spatial relations or counting. Adding a scaled vector at a selected layer during multimodal inference can improve visual-task accuracy without training on images or updating model weights. Earlier author manuscript 05.04 Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and BottlenecksPeer-reviewed ACL 2026 long paper. The study decomposes multimodal in-context learning into constructing a task mapping from demonstrations and transferring that mapping to the query. The tested MLLMs often form a useful mapping in middle layers, yet the signal weakens before late layers produce the answer. This suggests that weak multimodal ICL is not only failure to read demonstrations; it can be an internal information-transfer bottleneck. The proposed MGI intervention reinforces demonstration-derived information at selected layers and yields modest gains. Author manuscript · Official analysis code Work: Use matched behavioral, counterfactual, locality, persistence, and cross-modal interventions to compare localized unlearning, causal activation steering, text-derived multimodal steering, and multimodal in-context-learning bottlenecks.
Exit evidence: Report target suppression or steering, paraphrase and modality transfer, unrelated-task damage, relearning or extraction attacks, causal controls, layer/token sensitivity, and seed variance.
Research branches
Token attribution
Which input units influence which output under which baseline?
A Unified Approach to Interpreting Model Predictions · Axiomatic Attribution for Deep Networks
Mechanistic localization
Which components causally mediate a behavior across prompts and contexts?
A Mathematical Framework for Transformer Circuits · Locating and Editing Factual Associations in GPT · Sparse Autoencoders Find Highly Interpretable Features in Language Models
Knowledge editing
Does a change generalize correctly while remaining specific, persistent, and reversible?
Locating and Editing Factual Associations in GPT · Mass-Editing Memory in a Transformer
Machine unlearning
Is target capability merely suppressed, or is recoverable information reduced under a declared attacker?
The WMDP Benchmark: Measuring and Reducing Malicious Use with Unlearning
Localized unlearning and cross-modal activation mechanisms
Does an intervention remove information, alter one behavior, or expose a bottleneck—and does the mechanism transfer across a multimodal descendant?
Maximizing Local Entropy Where It Matters: Prefix-Aware Localized LLM Unlearning · CoSToM: Causal-oriented Steering for Intrinsic Theory-of-Mind Alignment in Large Language Models · Textual Steering Vectors Can Improve Visual Understanding in Multimodal Large Language Models · Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottlenecks
Comparison table
| Family or stage | Scientific distinction | Main risk | Anchors |
|---|---|---|---|
| Input attribution | Maps an output score to token or feature contributions. | Baseline and correlated tokens make attribution non-unique. | 2017 · 2017 |
| Circuit / feature analysis | Proposes internal computational units and tests interventions. | A located component may be neither sufficient nor complete. | 2021 · 2024 |
| Weight editing | Changes parameters to alter selected model behavior. | Collateral changes, reversion, and sequential interference can be hidden. | 2022 · 2023 |
| Unlearning intervention | Optimizes away selected hazardous behavior while attempting to retain general utility. | Benchmark suppression can fail under relearning, paraphrase, or stronger extraction attacks. | 2024 |
| Localized unlearning and cross-modal steering | PALU localizes entropy-based unlearning; CoSToM causally traces and steers ToM-related states; textual vectors probe cross-modal steering; multimodal-ICL analysis tests inherited language-model bottlenecks. | Behavior suppression is not certified deletion, steering success is not a faithful semantic circuit, and shared ancestry does not prove the same mechanism operates across modalities. | 2026 · 2026 · 2026 · 2026 |
Concept lineage
A Mathematical Framework for Transformer CircuitsLocating and Editing Factual Associations in GPT
Component-level decomposition becomes a causal localization and direct weight-intervention workflow.
Locating and Editing Factual Associations in GPTMass-Editing Memory in a Transformer
A targeted rank-one factual edit is extended to many associations and layers.
Mass-Editing Memory in a TransformerThe WMDP Benchmark: Measuring and Reducing Malicious Use with Unlearning
Targeted knowledge modification broadens into a threat-model-dependent attempt to reduce hazardous capabilities while preserving utility.
Failure modes to audit
- Naming a feature without testing its causal role.
- Evaluating edits only on the exact prompt.
- Ignoring unrelated capabilities and distribution shift.
- Treating model deletion or unlearning as established without attack tests.
- Inferring human-like concepts from a few activating examples.
Questions for every paper
- What behavior is localized?
- Which intervention tests causality?
- How broad should the edit generalize?
- Which side effects and persistence are measured?
Complete maintained paper route
Section titled “Complete maintained paper route”11 canonical readings · 2017–2026· route reviewed through October 2, 2026.
This is the complete structured list for this chapter, ordered chronologically from earlier foundations toward current work. The explanatory ladder above highlights the shortest conceptual route; this list also includes important branches, benchmarks, and current endpoints. It is generated from the same deduplicated catalog used by the AI background registry and My Library, so a future catalog update appears here automatically.
Show all 11 readings in maintained order
A Unified Approach to Interpreting Model Predictions
Scott M. Lundberg and Su-In Lee
Official artifacts: Official implementation
Metadata checked
Axiomatic Attribution for Deep Networks
Mukund Sundararajan, Ankur Taly, and Qiqi Yan
Metadata checked
A Mathematical Framework for Transformer Circuits
Nelson Elhage et al.
Author-published technical article; not presented as a peer-reviewed conference paper.
Metadata checked
Locating and Editing Factual Associations in GPT
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov
Official artifacts: Official project and code
Metadata checked
Mass-Editing Memory in a Transformer
Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau
Official artifacts: Official code
Metadata checked
Sparse Autoencoders Find Highly Interpretable Features in Language Models
Hoagy Cunningham, Aidan Ewart, Logan Riggs Smith, Robert Huben, and Lee Sharkey
Metadata checked
The WMDP Benchmark: Measuring and Reducing Malicious Use with Unlearning
Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew B. Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisa, Bhrugu Bharathi, Ariel Herbert-Voss, Cort B. Breuer, Andy Zou, Mantas Mazeika, Zifan Wang, Palash Oswal, Weiran Lin, Adam A. Hunt, Justin Tienken-Harder, Kevin Y. Shih, Kemper Talley, John Guan, Ian Steneker, David Campbell, Brad Jokubaitis, Steven Basart, Stephen Fitz, Ponnurangam Kumaraguru, Kallol Krishna Karmakar, Uday Tupakula, Vijay Varadharajan, Yan Shoshitaishvili, Jimmy Ba, Kevin M. Esvelt, Alexandr Wang, and Dan Hendrycks
Official artifacts: Official benchmark and RMU code · Project website
Metadata checked
Maximizing Local Entropy Where It Matters: Prefix-Aware Localized LLM Unlearning
Naixin Zhai, Pengyang Shao, Binbin Zheng, Yonghui Yang, Fei Shen, Long Bai, and Xun Yang
Peer-reviewed ACL 2026 long paper and Outstanding Paper award recipient.
Official artifacts: Author manuscript · Official code
Metadata checked
CoSToM: Causal-oriented Steering for Intrinsic Theory-of-Mind Alignment in Large Language Models
Mengfan Li, Xuanhua Shi, and Yang Deng
Peer-reviewed ACL 2026 long paper and Best Theme Paper award recipient.
Official artifacts: Author manuscript · Official code
Metadata checked
Textual Steering Vectors Can Improve Visual Understanding in Multimodal Large Language Models
Woody Haosheng Gan, Deqing Fu, Julian Asilis, Ollie Liu, Vatsal Sharan, Robin Jia, and Willie Neiswanger
Peer-reviewed ACL 2026 long paper; catalog authors follow the final proceedings record.
Official artifacts: Earlier author manuscript
Metadata checked
Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottlenecks
Yu Wang and Sharon Li
Peer-reviewed ACL 2026 long paper.
Official artifacts: Author manuscript · Official analysis code
Metadata checked