It’s more that it’s a very parallel architecture than fast.
The LPDDR5X under the hood is slower than what you’d find in a graphics card vram of years ago
That’s why you get consumer macs with 512gb while GPU makers are reluctant to give you more than 16gb unless you pay dearly. It’s not the same kind of mem
So not as flexible as apple's unified memory.