If there's a linear relationship between the weights and the output, it's not so hard to read the weights and reason about what's happening. If you read the reasoning traces for how the models do those tasks, you'll see that they rely on those direct relationships.

The same could never work in a larger network. It's not about just capacity to feed in the weights, it's about the complexity of the relationship. There's no demonstration here that an LLM can do that, and it doesn't really make sense to want it to. It makes much more sense to author code to analyse and modify than it does to do it directly.

I doubt this is either unexpected or emergent behavior. Anthropic and OpenAI are explicitly racing to reach RSI and I would expect them both to have trained their models on this problem, in hopes that it would generalize.

Yeah, I can read the weights in a small neural network, and for a given input, given that I know the activation functions, I can figure out what it does with pen and paper and lots of spare time.