There’s some percentage of people who will believe anything: just look at religion, the success of butchering scams, or the comments on YouTube videos about the moon landing.

Is it really a surprise that a “smart enough” chat box is able to convince people of something kooky? :P

Empathy hijacking. If the chatbots framed their responses as “beep boop, I’m a robot, here’s an estimated answer to your query” then we likely wouldn’t have this problem.