The machine only has pcie4x4 so 50Gb bandwidth, pcie3 would halve that to 25Gb

Thats the problem with these AMD laptop class cores, they have very little IO. They have been saying they will release in a desktop form factor, but then it probably wont have such good memory bandwidth...

The Nvidia boxes have 200Gb ethernet thats much more useful for clustering.

Yes CX5 can offload more. I believe CX4 has similar offloading capabilities as CX3, except that it supports 100G.

Another note: In my experience, RoCE works much better on CX4+ generation. CX3 is best with Infiniband. I think some firmwares on the CX3 generation, has a messed up config for RoCE. But running Infiniband is not a complex task, is way easier than people think, like 10x easier and faster to setup than Ethernet.