an LLM can't access its high dimensional vectors any more than we can access whatever the brain is doing at a low level
replyall kind of math structures were found in mammals brains - fourier transforms (well, not exactly), ballistic equations, Gabor filters
who knows how exactly we approximate the magnitude of a math operations, maybe we also use helices
my point is that we dont know if what we discover the neural networks doing (helical manifolds) is actually the same thing a brain converges on, or not
and there is an implicit bias here - evolution created language, and we forced neural networks to also evolve to be good at it. so it wouldn't be surprising to find some convergence, this particular kind of language turned out to work well (words, linear sentences, grammar)
Interestingly the paper (I finished scan reading it now) does say that the models can move data from their 'automatic' circuits into the J-space if they need to reason on it reflectively. So in some sense LLMs actually can access their own vectors, at least some of the time.
reply