I think the way it works is that the weights have to get streamed through the compute and the weights can be split very easily. With multiple cards you do have a coordination/communication phase after each layer of weights but that's usually much less significant than the compute phase.

It depends on how you slice the weights. You can split along layers which means each GPU waits for the previous one to finish or you can split the tensors across gpus.

Splitting the tensors is closer to n times the effective vram bandwidth but there are still overhead, sync waits, and reduce steps that eat away at the overall performance.