each adb host has to be individually white-listed by an unlocked device. also the current behavior is that it auto forgets any white listed host that hasn't connected within 7 days.
So even when adb is on an attacker can't just plug into your phone and use it. Besides, I just switch it off when I don't use it