Imo the counter-argument is that in principle you can research and understand the mechanics of climate change or medicine. You just don't have the free time and/or the motivation. With AI that changes - you could spend your entire life trying in vain to understand its reasoning.

It reminds me of how some religious people say that science is effectively no different to religion because we all take expert opinions on faith. But the difference is that there is a well-defined pathway to understanding, if you wish to do so.

This seems to me like asserting everything in the universe is explicable by physics. It may be technically true, but still not relevant to understanding earthquakes.

(Don’t bother to argue this not true unless you disagree with the essence of the argument.)

Anyway, post-hoc explicability isn’t a counter-argument to the assertion that almost everyone takes almost all technology as magic, from medicine to computers.

I’m still trying to understand your argument. Are you saying that after the fact we understand AlphaGo move 37? But somehow we are never going to understand an LLM’s decision afterwards? Seems like a disconnected take to me.