At the end of the day, even if they are some insane oracle (pun intended), they're still bounded by training data and how it relates to the real world. Even if they're a near perfect tool, we are still the interface between them and our lived experience. If that stops being the case then why do we care about the output?

This assumes it doesn't graduate to just killing all of us and doing it's own thing, but within this paradigm it doesn't really have goals.