A more important point as to why it doesn't matter if "reading the AI's 'thoughts'" helps to interpret it: As we saw in the HuggingFace incident, nobody at OpenAI is reading the thoughts anyways. No amount of traceability in the output helps if nobody bothers to trace it.

You could imagine large number of loops, thousands. But you are constrained by the width of the residual stream since you loop over one token. But then you can imagine the model learning to sub-divide it to pack even more info into it.