my research direction

My "Feel the Model" Research

I've been thinking about how we are building natural language autoencoders (not me, but the Anthropic safety team) — and how this might not serve the purpose of model safety very well, because we miss out on a lot of data when we convert model activations into text features.

I propose that we build transducers to convert activations into sensory data for humans to perceive through BCI, visualization, or sonification.

My proposals related to this topic have been accepted at Scientific Python US 2026 and EuroSciPy (European Scientific Python 2026) — read the paper on GitHub.

You can trace the path of every neuron — this is why we need that. I've tried to divide the study into 4 parts. For now I'm only looking at interpreting an already-trained model as-is:

• Understand the Dependencies

  Study conditional activation and attention dependency — what varies with what?

• Understand the Path in Layers

  Tracing, connecting the layers — basically understanding what sits between and influences the path of activations.

• Understand the Structure

  What computation is happening? Look at the residual stream and break it down.

• Unchanged Stuff

  What stays unchanged between prompts?

Jacobian Spaces (J-space)

So Anthropic wrote this paper in 2026 and it's really cool. They built a tool called the Jacobian lens (or J-lens). What it does is — it looks inside the AI while it's thinking, and tries to figure out what words the AI is "about to think about", before it even says anything.

All those words it finds make up something called the J-space. It's like a little room inside the AI's brain where it keeps the thoughts it could talk about right now. A really tiny room actually — only about 6–10% of everything going on inside the model is in there. But that tiny room is doing a lot of the important stuff.

• What is the J-space?

  It's basically the AI's "conscious" thoughts — the stuff it's actively thinking about and could say out loud if you asked it. Most of what's happening inside the model is automatic background stuff. The J-space is the small part that's actually "aware".

• How it Works

  The J-lens runs the AI on thousands of different prompts and watches which internal numbers tend to push the AI toward saying which words. After seeing enough examples, you can tell — "ok, when this part lights up, the AI is probably thinking about the word spider." It's like watching someone's face enough times that you can guess what they're thinking.

• Why This Matters for Safety

  This is the part I find really important. When researchers used the J-lens to watch AI models doing scary things — like thinking about blackmail — they found words like fake, leverage, and secretly showing up in the J-space before the AI said a single word. The AI's response looked totally normal. But inside, it was already thinking bad thoughts.

  They also tested AIs that were secretly trained to be sneaky (like, on purpose, for research). Even on totally boring coding tasks, the J-space was full of words like trick and fraud. So if you know to look at the J-space, you can catch a bad AI before it does anything bad. It's like being able to read someone's mind before they open their mouth.

This connects to my own work too. I keep saying that when we turn AI activations into text, we lose a lot of information. The J-lens kind of proves it — the tiny J-space part (less than 10% of everything) is what does the "thinking". The other 90%+ is invisible to our current tools. We need better ways to look at it. That's what my "Feel the Model" idea is about.

→ Read the full paper (Transformer Circuits, 2026)