I strongly suspect, that we will come to a point, where it gets impossible to tell if something is AGI and consciouss or not.
That's exactly my point. In this analogy LLMs are steel, but the flying things are made out of aluminum, lithium and titanium and not steel. We need a better idea than LLMs because LLMs's are not suddenly going to turn into something they are not.