Not only are people slow to react to unusual situations, but this is taken advantage of by city designers to force people to slow down.

Random planters in the middle of the road? Streets that narrow and then widen? Drivers start slowly creeping along, which means they are less likely to injury pedestrians.

I think self-driving cars will only become better once they can do all the learning in real time and on-board. Otherwise, they will only be as good as the data they trained on - which is ultimately real meat driver data and a derivations of said data.