Training is compute bound.

If its memory bandwidth bound you need to raise your batch sizes to leverage more parallelism.

It's also useful for prefill in inference, VLLM sees up to 30% boost depending on the exact kernels and prefill size