undefined

points

[-]

They must, right? They literally have no mechanism for cognition other than transformation of vocabulary. Language models are models fitted from data generated by humans, but they are not humans. Humans generate data by whatever processes happen in our brains, and LLMs of various architectures can learn a surprisingly good approximation of that data-generating process. That doesn't mean they have all the same characteristics and properties. That's true for all models really: model characteristics that are not captured in the goodness-of-fit metrics are not guaranteed to match the original phenomenon being modeled.

by mike_hearn1 days ago|

parent|

[-]

My interpretation of prior mechanistic interpretability research is that LLMs transform concepts in non-verbal spaces as well, which is why that was a surprising statement. For example showing how they do arithmetic on line lengths by rotating helical manifolds.

And the way they transform data isn't by transforming words. The layers transform high dimensional vectors - a format very alien to us. It's not obvious that these vectors must encode concepts from the vocabulary.

Edit: the paper claims that it's only J-space concepts that need to map to English words, other forms of cognition that are more 'practiced' and don't require so much reasoning bypass the J-space and can work in non-verbal subspaces. So that's the answer.

by rootatixww31 days ago|

parent|

[-]

an LLM can't access its high dimensional vectors any more than we can access whatever the brain is doing at a low level

all kind of math structures were found in mammals brains - fourier transforms (well, not exactly), ballistic equations, Gabor filters

who knows how exactly we approximate the magnitude of a math operations, maybe we also use helices

my point is that we dont know if what we discover the neural networks doing (helical manifolds) is actually the same thing a brain converges on, or not

and there is an implicit bias here - evolution created language, and we forced neural networks to also evolve to be good at it. so it wouldn't be surprising to find some convergence, this particular kind of language turned out to work well (words, linear sentences, grammar)

by mike_hearn1 days ago|

parent|

[-]

Interestingly the paper (I finished scan reading it now) does say that the models can move data from their 'automatic' circuits into the J-space if they need to reason on it reflectively. So in some sense LLMs actually can access their own vectors, at least some of the time.