It's worth pointing out that so many people already see and use AI the same way. In such a situation some would ask AI to provide options, and they would choose and experiment with those options. Of course, contexts/stakes can be vastly different.
Given that there are so many phenomena in nature that we can't explain or fully understand which doesn't prevent them from existing or being useful, there could be a future where humans accept the same about the things AI comes up with as long as this leads to desired outcomes. We still might have names for them, but our brain thinking/knowledge capacity wouldn't allow us to fully comprehend them. We'd need frameworks/systems in place to turn these AI features/decisions on/off, although it's hard to imagine how this wouldn't increase the likelihood of something going out of control.
For it to have new vocabulary we dont understand, it needs to have novel ideas that need words coined for them, and a way to persist those ideas and words into the future. I don’t think that exists.
To me this hypothetical make it clear this won’t happen, not unless there are fundamental changes to what llms are. It doesn’t suggest it will happen. To me, anyway.
Theoretically you could update the weights at inference time too though so the model evolved as it's used. Surely some people are trying this already.
But given our current relationship even if it did I can't foresee a point where it couldn't walk us through the necessary steps or supply the pros/cons for whatever problem is being addressed.
2 issues - trying to understand how it came to its conclusion because I feel true AI has got to be non-human intelligence. Or a something catastrophic happens and we as a species are back in the stone age. Imagine today's AI trying to converse with a cave man (yes, one without modern languages even).
We can think of questions we can't answer. It can answer them.