The other part of it is that the MSRP already baked in a substantial increase from the previous generation. While RAM was near rock-bottom pricing when this hit, current-gen GPUs definitely were not.
It might also be that NVIDIA is a natural monopoly, while memory manufacturers are a cartel...