If you really want to see why all the cruft is there, and how much you could remove if you dropped support for older hardware, look no further than Sebastian Aaltonen's "No Graphics API" blog post (which he actually implemented too): https://www.sebastianaaltonen.com/blog/no-graphics-api/
E.g. if CPUs would have closed ISAs and heavily guarded IPs, you'd also need a 'driver' and an 'OpenCPU API' to make any use of them (of course the public ISA of a CPU is essentially the driver interface baked into the hardware).
The other big difference is focus on decent single-thread performance for CPUs versus performing thousands of small tasks in parallel on GPUs, but each one slowly. This difference causes a different programming model (e.g. running an entire application in a single 'main' function from start to finish, versus spawning small specialized tasks which are running across hundreds of threads).
And the specific problem with Vulkan is that it tried to solve two conflicting requirements: on one hand being an explicit low-level API without 'magic abstractions', on the other hand supporting very different GPU architectures (especially the mobile GPU zoo). The resulting mess was quite predictable to be honest...
A bunch of the complexity comes from having to write both client and server, and the communication protocol between them, then some more comes from the client being responsible for spinning up the server. As with every network service, client/server traffic is the slowest part of the architecture, so a bunch of complexity then comes from having to minimise the amount of network calls (or draw calls, in this case).
Then you hit the part where all of this is performance critical (soft-realtime requirements), so the whole thing is geared towards “writing your own userspace IP stack in C++” rather than “writing a flask application in Python”.
CUDA being the winner (closed source, not really usable on non-nvidia GPUs) between those 2 and nvidia not wanting to open their competitive advantage to the competition for the niche of non gamings uses / scientific computing (and later AI, which became a huge win for nvidia).
vulkan came around after as this open thing (based on a previous attempt by AMD called mantle) that's trying to break down the leaky abstractions and the divide between language for graphics & video games and separate language/frameworks for gpu non-graphic computing with the promise of giving more control and more room for squeezing performance from the hardware at the cost of more complexity with the idea that you'll probably build your own abstractions on top of it for your subset of applications.
If GPU were from the get go designed as a parallelism tool first and video game tools second the hardware architecture and the language would probably be different than today and simpler (and if the competition between the duopoly didn't sabotage things)
don't forget that the drivers for GPUs would until recently get updates to patch specific video games for performance issues (not sure if they still do that ?)
Sony tried that with the PS3's Cell processor, just a wise parallel vector processor instead of dedicated hardware.
If you knew how to use it, you could get great results, which is how Naughty Dog's games became so spectacular. But if you just tried to port an existing game, things went downhill quickly, which is why GabeN talked on stage how much he hated the PS3.
Oh and this complexity led to lots of games just switching to one of the existing engines that supported it natively, and started the modern dominance of Unreal Engine. But it's also why engines introduced more parallel architectures, which a few years later became very useful when quad-core processors in PCs became mainstream.
The market demand was for video game tools, though.
People didn't wanted faster compute, they wanted their computer games to work smoothly on the hardware they already own. Therefore, they preferred to spend 100-200€ on a point upgrade, much like RAM, than scrapping their 2k€ PC.
To give people a reason to buy these upgrades, games had to use them. Therefore video cards were designed specially for the usecases and development flow that game developers already followed.
If it wasn't for the video game market, NVidia wouldn't exist as we know it today.
There are a few libraries that try to abstract over all this indirection but all the graphics APIs themselves are lower-level so you're left to do all of that yourself.
But GPUs have thousands of cores, which means if you want any benefit, the smallest data type you must use is something like array[n * 1024], or your GPU is wasted doing nothing. So you must shape your programs into that memory model or you're just wasting your time (and GPU).
That's >50% of the problem.
Some OS/hardware have separate GPU memory (Nvidia), some have unified memory (Macos, modern consoles), but moving memory is usually not the problem.
Things don't "run on CPU by default", it's up to the programmer where to run it. But the CPU and GPU paradigm of "running a program" are very different, since on GPU you're running SIMD (i.e. running your code on arrays) or else there is no point to use a GPU.
the key for me is thinking in terms of sets—much of your conditional logic is probably about sorting things into sets anyway.
But also partly in the fact that the GPU is specialized for certain operations over a CPU. It wants and need to obtain data in a different way from a CPU to maximize throughput. So language constructs are designed around making sure developers stay on this fast path. The complexity can be seen as some intentional gatekeeping to signal that this kind of programming is only for specific kinds of problems.
there's HW references + the linux drivers for AMD (both UMD and KMD), there's refs for NV and their open linux KMD (no UMD though), intel also has HW refs and open drivers for linux
I think it would be possible to put together as long as you don't mind only working for certain hardware, like a prototype you know
I might give it a shot to make a proof-of-concept "high level low level" graphics API ngl if no one else feels like so...
I think there's space for a "two-level" design like have the driver consist of the HW-specific parts, expose things in terms of GPU VAs, explicit sync, explicit memory management, command buffers etc. then have a high-level wrapper over it to provide vaguely GL-ish semantics, and you could intermix the two APIs, obviously you wouldn't be able to get the VA for a high-level object because that'd break the autotracking, if you use low-level you're responsible for syncing, cycling etc.