Techniques | Model Recon

Things I'm Building

in progress
Circuit Heat Maps

I am building models with the sole purpose of drawing circuit heat maps and flagging issues in the model under monitoring.
in progress
mr-recon-tracer

I am building an array of visualization tools for analyzing the activation paths. Reduced data extraction is necessary to do that. This tool will extract data in my proposed Activity Cube format. → github.com/modelrecon/mr-recon-tracer

Here are some techniques I've been reading and thinking about, in rough order of my interest in them.

Mechanistic Interpretability / Circuit Tracing & Sparse Decomposition

This is my favourite one. It is a set of techniques that attempt to peer inside deep networks (especially large language models or transformers) and decompose them into simpler, interpretable sub-components — e.g. "circuits," "features," "concepts" — rather than treating them as opaque black boxes.
LIME (Local Interpretable Model-agnostic Explanations)

This is the second fav. It creates a surrogate model trained on input/outputs of the core model:

Takes one input instance → Creates many small changes around it → Gets predictions from the black-box model → Fits a simple model on those perturbations (the surrogate) → Uses the surrogate for the explanation. Pretty simple!
Shapley Values

It is weird — this was a technique used in 1954, so old! It is a game theory concept applied to AI. I don't understand game theory very well, but what I know is that it takes a feature and finds out how much it attributes to the output by giving it values of different kinds.
ViTmiX

This one is just for image models. It is a mix and match of various techniques. Good enough for now.
XAI‑Guided Context‑Aware Data Augmentation

It is a performance improvement technique for data — it may not be great to consider as a pure interpretability technique. What it does is iteratively make the least important features more important by changing tokens.