It's GPU drivers freaking out when they run out of VRAM, but it might only be specific to NVIDIA cards since AMD seems to handle this better.
But, if it's driver specific that means it could be fixed at the kernel level I suppose, especially since Windows handles it seamlessly?
I suppose this could be an historic thing, GPUs can have a type of DMA now, and disk access is much faster too, so paging perhaps makes sense when it didn't before?