Now the idea of AGI has been narrowed and scoped to economically viable work. Even Turing had a different idea when he asked "Can machines think?".
Now programmers and mathematicians are being superseded by AI, both professions long deemed the pinnacle of human intelligence. Somehow, now plumbers occupy that spot.
How is "people not knowing what consciousness is" relevant here in the first place? AI already can do practically everything the human brain can, and often better or at least faster. The "tipping point" arguably isn't only close, but we're practically on top of it.
You evade the crucial point in any case: the lack in ethics and empathy is far too prevalent in humans already, but has certainly never prevented them from doing harm.
People somehow forget that the original Turing Test was designed to compare two participants chatting through a text-only interface: one AI and one human. The goal was to spot the imposter. Today, the test is simplified from three participants to just two: a human and an LLM. This changes the test from a comparison to a judgment.
Stop spreading misinformation and partial truths!
The Turing Test was to figure out which it the participants was a _Woman_ not human!
What really matters are the core aspects of intelligent behavior. Pattern recognition, planning, adaptation, etc.
It really doesn’t matter if an intelligent system is conscious, or how similar it is to commander data, or even how much economically viable work it can do.
It means AI that is General, as in it is not specific to one narrow task, like object recognition or playing chess.
This was a hard problem for decades. No AI was general, until GPT 3 or 4. Now we have General AI.
So we have AGI.
GPT6 will attempt to do almost any problem you can give it in text or image format and it will actually do a decent job a lot of the time. But its performance is still extremely spiky and it still makes basic mistakes and hallucinations.
So it's definitely a general artificial intelligence in some sense but it's kind of a weird one compared to the classic scifi idea