Let's start with grouping messages into conversations reliably. Whatever black magic they are currently using is almost there, but has it's flaws.
NewEmailServers are able to send and receive messages using either the Email or the NewEmail protocol.
Before sending a message, the NewEmailServer checks a directory (similar to DNS) to see if any of the recipients are on another NewEmailServer. If so, the message is sent to those recipients via NewEmail. Others get it by Email.
Completely backward compatible. Once google/microsoft shift to NewEmail, the rest will follow quickly except some legacy servers. All that is needed is a standards body.
When dealing with any kind of network effect and existing user behavior, though, especially one as ingrained as email, I would highly recommend being judicious about using the word "just," given that these are mutually-reinforcing effects. Imagining a solution is a world away from implementing one.
Still, I think you're on the right track: create a new protocol on top of an old one, use the same input methods and user behaviors, call attention to when someone is not using it (e.g., green bubbles), and make it extremely easy for existing services to migrate to use it (i.e., the standard/standards body, notably NOT what Apple has done). One part that's missing here is some pressure to make this happen: positive pressure (economic, product sense, persuasion) or negative (economic, social, regulatory).
The stories have discouraged my from spending my time on it though.
However, most of time, it's not that providers are blacklisting because "Ha ha ha, screw the little guy" but more "Yea, they are hosting on IP blocks with terrible reputation and spammers are absolute liars. I refuse to trust whatever they say, period."
If you have your own /24 or greater, you can become email provider. It's also just such a low margin business, it's not worth small businesses disrupting it.
I blacklisted pretty much all of africa and arabic nations and china on the IP level, ( ensure you get the bgp routed datacenters in those nations too) and not only did i get less spam, but less automated bot attacks on my webserver.
The reason people talk about the blacklist is because its just so damn effective.