https://dnhkng.github.io/posts/sapir-whorf/
The middle layers also perform reasoning on the abstract concepts, to the point that you can replicate some blocks of inner layers (thus giving the LLM more internal "reasoning space") and by this increase the model's reasoning abilities. The video in this article shows that when performing a sequence of arithmetic operations (without CoT, i.e. the result is spit out directly), internally the intermediate calculations are spelled out, and this can only happen in the depth direction of the LLM (since no new token is added to the sequence). So this "jspace" can only be situated in the middle layers, probably in circuits that repeat nearly identical across several layers.
The finding that you can repeat the middle layers pairs neatly with Anthropic's finding that there is some internal CoT-like process happening in them. I'm not sure how to find those blog posts, but maybe someone else remembers them
> Recent research on smaller models has shown hints of shared grammatical mechanisms across languages. We investigate this by asking Claude for the "opposite of small" across different languages, and find that the same core features for the concepts of smallness and oppositeness activate, and trigger a concept of largeness, which gets translated out into the language of the question.
It's still not clear to me intuitively what this represents though. I get that it somehow encodes a link between future words the model says and the current activation, but the confusing thing is that I always think of derivatives and gradients as basically a "sensitivity" between output & input, i.e. if you nudge the input x by h, the output changes by h * f'(x). So then on the face of it applying the J-lens matrix directly to a given activation rather than a small perterbation seems like a "type" issue.
Maybe we're somehow treating f(0) = 0 so that you can apply it directly? Or is there some shift invariance somehow? ignoring that, I do see how it's like selecting a linear combination of the directions, and then it can maybe be represented as "possible continuations" in the same way the gradient is usually thought of as tangent space. Maybe that's what the other commenters meant by information geometric approach.
Other things i'm not clear about is how this is related to two other interprability methods: * SAE (sparse auto encoder) they showed a few months back, where you train an autoencoder directly off of the hidden states/residual stream to convert it into words. The doc only mentions it briefly, but it seems that j-lens is sensitive to things that SAE are not. They're both working off of the same residual stream so clearly the the inputs must be there, but for some reason SAEs can't detect it while J-lens can (they seem to hint at some explanation but it's over my head)
* Logit lens. This was a more primitive technique that simply applies the unembedding matrix directly to the residual stream. I do like that they mention it:
>The J-lens can be understood as a principled refinement of the logit lens. While the logit lens assumes that representations use the same coordinates in all layers, the Jacobian lens corrects for representational changes that take place across layers, allowing it to uncover meaningful information in earlier layers where the logit lens produces uninterpretable readouts... The J-lens can be understood as the principled correction: J_l is precisely the average linear map that relates layer-l directions to their final-layer counterparts.
Their method is used to identify which tokens can appears in which layers of the model.
I too have confusion.