Evolved systems need no high order understanding of how they work. You have evolution take care of that hard work for you, hence why these models take exaflops of compute and gigawatts of power to train.

Understanding and accidentally creating are different things. If we already understood consciousness and weren't worried about having accidentally created one already, we won't be having this discussion. If we don't know what consciousness is, in my view its more reason to exercise some caution regarding a system that appears to exhibit intelligent thinking, talking, etc.