Now, if the threat scenario is someone implanting a compromised version of the IM app on every device out there, and siphoning data from the device itself, then it's a completely different scenario.
[1] although this could be intercepted by an attacker compromising the IM servers, if the app is not distributed/P2P
You or the other person could lose the device and someone could use your PIN/password (something as simple as shoulder surfing while you use it). There could also be a leak in whatever cloud service you're using, or the data could get subpoenaed because of some dumb law that gets passed, some rogue employee, etc. It's a huge liability no matter how you look at it.
But for normal people, the biggest risk is companies using their chats to train models / dispatch ads etc to which the only solution is E2EE.