You or the other person could lose the device and someone could use your PIN/password (something as simple as shoulder surfing while you use it). There could also be a leak in whatever cloud service you're using, or the data could get subpoenaed because of some dumb law that gets passed, some rogue employee, etc. It's a huge liability no matter how you look at it.
But for normal people, the biggest risk is companies using their chats to train models / dispatch ads etc to which the only solution is E2EE.