But how do you know it's not conscious? It's a very poorly defined concept.

I know various people that to this day say that fish do not feel pain (because they want to catch them with a hook through their mouth without feeling guilty). That seems a ridiculous notion to me, as pain is extremely evolutionary useful and a fish displays all sorts of pain-like behaviour when hooked. But still, since we can't really look inside the fish' mind people can make themselves believe they don't feel pain.

If you ask the right AI if it's conscious, it's very well possible it will say yes. Because it was trained on the world literature maybe and behaves as learned. Is there a difference with us? I'm not so sure.

To me it's kinda weird the ethical implications for striving for AGI are so little talked about.