The downside is never really knowing if your email reached your recipient (maybe it got blocked by Gmail because of your IP, maybe some other reason). If I use Gmail/proton etc, then I can be pretty sure that the other person got my email.
It's not trivial to run your own email stack, but it's not hard either. It's more tedious than anything. Once it's up and running, it's low-maintenance. And it's fairly trivial to avoid spam. I get vastly more spam in my Gmail than I do on my self-hosted domains.
The biggest challenge with self-hosted email is getting an IP address that's not blocked or carrying low reputation. Most VPS IPs from e.g. Digital Ocean, Linode, Hetzner, AWS, etc. are going to get blocked from Google (the strictest) and Yahoo, Hotmail/Live/MS, etc. Cloud providers often block incoming and outgoing mail ports, too. I have a Digital Ocean VPS and my account is old enough to be grandfathered into the old policy, which means my email ports aren't blocked. I used to have no trouble sending email anywhere from my VPS, until I downsized it and got stuck with an IP in a bad reputation CIDR. Now I use smtp2go for relaying outbound mail, but still handle all inbound mail. 10/10 score on mail-tester.com, and I never have any problems or need to do any maintenance.