I simply use the port that is closest to the MagSafe. Haven't tried much with the M1 macs though, since because my M1Pro MBP was issued by the company and had lots of restrictions on it.
According to Apple, that's actually the DFU port.
In any case, Apple's docs don't say that you always have to use the same port with an external drive.