it's a good idea but you often end up with huge tail latencies, slight incompatibilities due to fleet heterogeneity, and a database/scheduler management problem.
https://developer.nvidia.com/gpudirect
And this is because I can't be bothered to dive into older papers, so recent NVIDIA stuff will do.