That is not how I remember it (for a data point of one shop in New England during the time): we embraced it because of the binary validation under multiple theories. There was a strong suggestion valid html did better from an SEO perspective, so we could sell that, a suggestion browsers would be less buggy with properly formed xhtml and a number of theories about what the future held for bots and scrapers to be able to easily ingest and parse your content (seen as a good thing then).

It failed because the smallest error by a client after the fact was like a server crash. Plus it would have created a mild barrier to entry when learning html at all.

> I think XHTML failed because it didn't give web devs any new capabilities,

and what new capabilities does this new proposal provide?