It is the same as speed limits. Very good or excellent drivers can drive much faster than the speed limit and be just as safe as the average driver at the speed limit, or a very poor driver going much slower than the speed limit.

These things are set so that, on average, everyone is safer. If allowing self driving cars raises the safety of everyone on average, it makes sense.

Personally, I would be in favour of everyone having to either use a self driving car or gain an advanced driving certificate so they could drive manually. One that only 10-20% of people could pass, this would include an in depth theoretical section as well as a 2-3 hour driving test.