agree reasoning fixed a lot of it. but the inherent path dependence of autoregression is one reason i was excited for text diffusion models (https://www.youtube.com/watch?v=r305-aQTaU0)

instead of going left to right, even with a scratchpad, maybe you start with a rough shape of the big picture all at once, and then you iteratively resolve and things come into focus.

mercury (https://www.youtube.com/watch?v=2fDBeMu6xjk) seems to have made the most progress here, which is not saying a ton but is not nothing. i do think it is telling that of the big labs, only GDM has made any meaningful bet on text diffusion. you can bet your ass all of them have evaluated it for a source of alpha.

In terms of our brains though we can only think forward as well (if forward is time). Our brain in the future says something we did in the past was wrong (part of the sentence we wrote) and that informs our body (the agent) to go back and fix it