To analogize, current transformers run a fixed-length program per step. Any program can be factored into a top-level loop with a fixed-length branching body (an interpreter). Dynamically looped transformers can run any program between tokens.
The safety argument for CoT monitoring is that in transformers information about the hidden state has to be communicated through the bottleneck of sampling a single token per forward pass. If not trained adversarially, it’s likely that a reasoning trace contains all the “bottlenecked information” we need to determine intent. But if we can compute arbitrary programs between tokens, the reasoning used is hidden.
It also opens the door to simple architectural extensions that would make the safety/monitoring side of things much more difficult.
It’s probably fine in practice at these scales though. If we keep each loop turn reasonable non-deep, we can probably recover most of the benefits by decoding “extended” CoTs from the residual stream at each loop turn between tokens. But that’s an area of active development.
The ability to compute any computable function between tokens given an ability to loop an arbitrary number of times is a nice theoretical point, sure, but ultimately if people are still using single digit hard cutoffs on the number of loops, I'm not sure it's all that important.
So, I agree it's right to say that arbitrary length dynamic looping could open the door to making monitoring very hard indeed, by extending hidden states further and further. But I would speculate that if it actually worked better than extending the sequence with CoT tokens, we'd already be seeing it in strong open weight models. It's a fairly obvious thing to try. And we're not seeing it, AFAIK. So I do wonder whether it's something we really need to worry about in practice, compared to all the other things we have to worry about.
In your generalized example I think the concern is when the additional evaluation effectively becomes a replacement for CoT, where something like the coconut research could replace it completely.
However, I don’t think we’re anywhere close to that with Astra.
Wrote about it here: https://substack.com/home/post/p-214402969
(For all intents and purposes given how high dimensional you are and using the "vibes" of computability yes I agree w/ you)
This is worded so confusingly it might as well tell us nothing, because it is technically true even without looping due to the fact that you still have infinitely growing context and can simulate a standard turing machine using it.
If you loop, you have a fixed capacity memory that you can rewrite but not carry over to the next token, this is different from a non looped transformer where the transformer can only append a new token.
Meanwhile if you have a DEQ with growing context, it is bona-fide turing complete in the most literal sense.
It's not a "scary new thing" but ultimately no-one knows exactly how OpenAI have implemented looping. You might not be aware/remember but MoE transformers perennially underperfomed their dense counterparts until GPT-4. Similarly, making reinforcement learning really work with transformers wasn't figured out until o1.
And by Open AI's own admission, Astra's CoT is significantly harder to monitor and it exhibits a significantly greater control over its own CoT than any other model released.
> And by Open AI's own admission, Astra's CoT is significantly harder to monitor and it exhibits a significantly greater control over its own CoT than any other model released.
Oh sure; I don't think anyone is denying that larger issue? But does it have anything to do with looping?
If the model has significantly more ability to stuff away information outside visible reasoning than every other model including ones in its size class then surely it is reasonable to assume the architecture tweak that allows the model to compute more before outputing a single token is somewhat responsible for this change ?
I guess we did manage to eventually seriously crash into the wall "more compute than normal (non-looped/unique-weights) transformers can efficiently consume with the limited training data we have", plus massive focus on highly hands-off agentic tool use reasoning...
https://arxiv.org/abs/2310.07096
Edit: read much of the article, it's brute force predecessor was explicitly called out as an almost-ancient example:
> The looped transformer is nothing new, and the basic idea already appeared in the Universal Transformers paper from 2018
This may be a bit of a nitpick, but... does it? I agree that giving the model decisions on looping certainly makes interpretability harder, because it adds more transient internal states to deal with and changes the number of them depending on prior states. But is it really pulling CoT inside the forward pass, if the sequence length it's operating on isn't growing? In some sense the whole technique and tradeoff of CoT is "add more tokens to the sequence, use them to reason with", with one of the benefits being, you force the model to output tokens, so you can (hopefully) understand it. And the big point TFA is making is, nobody is doing recurrence over sequence length as far as we know.