Sony tried that with the PS3's Cell processor, just a wise parallel vector processor instead of dedicated hardware.
If you knew how to use it, you could get great results, which is how Naughty Dog's games became so spectacular. But if you just tried to port an existing game, things went downhill quickly, which is why GabeN talked on stage how much he hated the PS3.
Oh and this complexity led to lots of games just switching to one of the existing engines that supported it natively, and started the modern dominance of Unreal Engine. But it's also why engines introduced more parallel architectures, which a few years later became very useful when quad-core processors in PCs became mainstream.
The market demand was for video game tools, though.
People didn't wanted faster compute, they wanted their computer games to work smoothly on the hardware they already own. Therefore, they preferred to spend 100-200€ on a point upgrade, much like RAM, than scrapping their 2k€ PC.
To give people a reason to buy these upgrades, games had to use them. Therefore video cards were designed specially for the usecases and development flow that game developers already followed.
If it wasn't for the video game market, NVidia wouldn't exist as we know it today.