This is hardly new or novel, you’ve just described huge chunks of existing engineering disciplines. There is still no complete model that explains why airplane wings work, there are lots of very useful models, but all of them have fundamental flaws where their behaviour completely breaks down in certain very possible scenarios. Notably scenarios where airplane wings don’t spontaneously either stop working or explode.
We also built the entire commercial airliner system decades before we even had computers capable of doing the aerodynamic analysis of airplanes, but none of that prevented us from creating huge engineering disciplines around the empirical data we did have, and slowly chip away at the underlying theories as maths and computing improved over time.
So we’ve always lived in a world where we extract value from systems we fundamentally don’t understand. But that’s never stopped us from working to understand them anyway, and deriving even further value from that improved understanding.
So far our solutions were more or less understood thru some models of reality that we’ve constructed (on our own), which may or may not reflect reality perfectly, and even if most of us never bothered thinking about these models, some people did and they understood them on a very deep level.
But we may be getting to a point where the problems we need to tackle become too difficult for humans to model, or even to notice their existence, like asking an ant how a Boeing 747 works.
Maybe this was always the case but now it seems like we might have a shot at making these solutions useful even if we have no idea what they’re even solving.
The author mentioned what the issue is: a lot of people are experiencing what it's like to 'catch up' to the more intelligent ideas. Everyone can 'get' the intelligent ideas given enough time. But how much time do you have? And he's concerned someone will rush and create some kind of world-ending solution because they're too trusting of a technology they don't understand.
His only solution is some kind of throttle on the advancement of technology, and arguably knowledge (whose?).
You observe others behaving poorly and immediately succumb to defeatism, assuming there was no better way.
When presuming, all that matters was the short term economic profit, you simply use the wrong gauge.
Here, long term human survival is at stake, regressing to zoo animals isn't a sensible option.
Do you think every Mathematician has a deep understanding of building architecture, engineering, physics, economy, etc., everything necessary to gauge whether something like a 1 terrawat nuclear plant is completely safe in all aspects?
Of course not, so then you have to select appropriate ones. And then you need a process to ingress and egress reviews. Oh what about change? Does that come back to the Math council too?
Oh, we also need global cooperation to pull this off. And then do you trust the people, the selection process for the council (all avenues for corruption are there).
I don't have a solution of my own, so it's not fair for me to criticize the author in this way. I share his concerns.
And it's depressing to think of all the negative scenarios, and be so concerned all the time.
These issues are important and worth thinking about, regardless. Who should teachers teach?
People used to solve logic problems by hand, verify logic. Now you pile it into a computer. The problem has been around for a while, no-one fully ‘understands’ the proof of the four colour theorem as a big chunk is computer proved.
We are now just changing (admittedly greatly) what we can put in a box marked ‘checked by computer.
People didn't stop understanding how to do addition or subtraction when the calculator came out. That's a simple example, but if the AI is superhuman, I don't see any reason why it cannot break things down into simple concepts. I don't think anything is truly beyond comprehension; it just needs to be explained properly (by someone, or someTHING that really understands it) and, for complex ideas, time taken to understand them.
> given up on understanding parts of our own systems for the exact same reason.
Maybe this will be a sign of when ASI is achieved. AI researchers keep saying they don't fully understand how LLMs work - maybe when ASI arrives, it can explain that fully.
You should try being a teaching assistant or a tutor. "Everyone can be taught everything, it just depends on the teacher" can only be said by someone who never tried this in practice. Just as not everyone can learn advanced math, the best mathematicians also have their own limits and there are things they won't be able to understand or comfortably navigate. Not just because it's a lot of material, but because it's complex inherently.