Because even CPUs rather use JIT runtimes to deal with the various kinds of ISAs that exist.

Naturally plenty of folks rather use software that doesn't take advantage of the hardware they paid for.

If i'm not mistaken, this already exists, and the assembly language here is called PTX

https://llvm.org/docs/NVPTXUsage.html

That would require vendors to either stick with a single backwards compatible ISA like intel did for x86 or document how their graphics cards work.

CPUs manage this by changing the internal micro-architecture, but historically GPUs only needed to support a graphics API and used that abstraction layer to freely change the hardware.