This is a strange attitude. When an agent is optimizing a piece of code, comes up with 2 variations, and runs benchmarks on them to figure out which one is faster, then selects one of them based on tradeoffs between performance and other things it reasons about, do you ignore its explanation and all experiment runs?

You're confusing the weights of a model and internal chain-of-thought with the output of the model. Yes, we don't know a lot about how the internal mechanisms work. But with the correct prompt, agents will produce a worklog that documents exactly what solutions were tried and how the result was obtained.

>Their thought process is effectively undecipherable by humans (it's essentially information arising from information

Are you trying to say that human brains are incapable of inference?

What are you on about? I have had Fable come up with new shit for me several times (I do research for a living, so actual new shit nobody knew before), and each time it was perfectly understandable.

Of course I don’t know how it got its ideas for what to try. But heck, I don’t even understand how I get my ideas half the time. But the process, like what code it wrote, simulations it ran etc can be understood by (some) humans just fine!

Yes I saw 3Blue1Brown say the same thing in his tutorial on how neural nets worked where he built a simple model to recognize a particular letter. Good reminder.