The pitfall you describe is not inconsistent with exceeding human performance by most metrics.
replyI'd argue that the current system in the west already exhibits this problem to some extent. Fortunately it's a systemic issue as opposed to a technical one so there's no reason AI necessarily has to make it worse.
That’s not really an argument, it is central to my point. The current system does exhibit those issues and it is by human creativity and outliers that we have some points of escape from it.
replyCodifying and distilling it removes the points of escape.