The downside is obviously that uncached queries take much longer (adding >100ms) and more queries are uncached since you can't share the cache with a large user-base. Unless you just visit the same websites over and over again, this results in worse overall performance.
the _one_ downside i've seen is on an airplane serviced by Starlink: UDP was extremely lossy to the point that whatever recursive resolver i was using at the time would mark half of all nameservers it saw as "unhealthy" and start returning NXDOMAINs to the clients before even trying to hit the authoritative NS.
Versus letting a singular entity snoop everything? If you actually open a connection to the result what is the difference? The only way to fully deal with all that is an overlay or mixnets.