"A novice was trying to fix a broken Lisp machine by turning the power off and on.
"Knight[, one of the principle designers of the Lisp machine], seeing what the student was doing, spoke sternly: 'You cannot fix a machine by just power-cycling it with no understanding of what is going wrong.'
"Knight turned the machine off and on.
"The machine worked."
I feel like AI is manifesting this even more concretely. I don't feel like I'm guiding the AI super intensely as I work on it with software engineering. I'd have a hard time pointing you at where in the prompt my decades of experience are manifesting. But I definitely can have better results, even with a less frontier-level AI, than people who don't know the same amount of stuff.
Terence Tao also released some unedited transcripts of some of his conversations with AI, and many people observed that while many mathematicians may have been able to formulate the initial question, very few people could have given the same feedback to the AI.
Perhaps someday AI will eliminate the need for competence to use it properly. But that day is not today. And to be honest, that tech is probably not LLMs, no matter how large they get. Some other breakthrough will be necessary to truly eliminate the human element. Those psychopathic elites making plans to turn Earth into one of the Spacer worlds from Asimov's works with a small elite population supported entirely with robots take notes... it's not possible yet.