The danger is in the replacement. Confiding in others builds a web of trust. Talking about a break-up with a friend builds a relationship, and the next time that friend needs someone, they know you're an option. It's economics, borrowing and trading emotional support. AI as confidant destroys that economy.
This line of reasoning doesn't hold up for many harmful habits that people voluntarily partake in. It's OK for someone to believe that using an AI as an approximate therapist might be a bad idea, especially in light of the many publicly documented cases of AI feeding delusional thoughts and encouraging suicide.