You and previous comment seem to agree that trial and error/shared experience can determine if consciousness has emerged. And this might/will be challenging.

Previous comment used the word "anticipate" and I think they mean that we won't know in advance before we run the trial and error process.

When they say "deep trouble" I assume they mean because creating a non-friendly conscious AI might pose an existential risk for humanity.

However there is also the ethical issue of creating a consciousness and then destroying/murdering it.