To be fair, Musk was only parroting what Karpathy was telling him so you should ask him how self driving cars are supposed to work with CV only.

Well he's also argued that just using CV reduces sensor contention and he claims it improves performance and release velocity, which is why they also got rid of radar and ultrasonic sensors. I am doubtful although it'll be interesting to see regardless