Because - it's the part people are having hard time to accept - creativity was never the hard part. It's randomness. The hard part is navigating the narrow border between "not enough random, therefore boring" and "too much random, therefore unhinged/insane".

That border is determined not as much by ground truth, as by people's sentiment. As such, many famous artists and creatives never succeed navigating - their contemporaries would call them crazy, and then decades or centuries later, the border shifts, and suddenly they're remembered as greatest creators of their era.

the GP posits the problem with "recursive self improvement" is the poisoned context & hallucination problem. While a strong loop that has fail safes, backups, restore points (essentially, a fancy backup system), the problem isn't that we cannot create a healthy advanced wiggum loop; it's that every step of the LLM as it grows whatever knowledge is acretes, has a chance of being either poisoned (eg, it conflates two tokens as describe different things) or wholesales fabricates a method or procedure.

Now humans are just as bad, but they're not moving at the speed of compute so the posion and fabricates can dissolve over time, or just, as you've noticed turning on your news, get stuck in very stupid positions. So humans are clearly capable but clearly don't tend to do this either.

So then we dont have a real road map. The error rates, although small, acrete at exponential levels and will wash out improvements.

So I also had the idea that "if we just give it enough context, surely it'll be more powerful". But the error rates hit that squarely. The larger the context grows, the more likely it hasn't properly organized its knowledge to avoid overlapping facts.

In programming, it's worse, because a lot of the code is purposefully "DRY" and reuseable. Everye C program has a main(); is it remembering the correct main? or any of the number of same variables?

You can see an LLM is powerful but it's not ominipotent. It'll suffer very much when it starts hallucinations and context poisoning.

So, sure you can try a super ralph wiggum loop with memory, fallback safeties, etc, but you basically then need another turtle that does the same thing, and at that point, you're positing a infinite jest of ralph wiggum loops tracking each other, recursively, forever.