I mean it doesn't. If you actually look at it comma.ai proves that level two doesn't require lidar. Thats not the same as full speed safe autonomy.
whilst it is possible to drive vision only (assuming the right array of cameras (ie not the way tesla have done it) lidar gives you a low latency source of depth that can correct vision mistakes. Its also much less energy intensive to work out if an object is dangerous, and on a collision course.
To do that in vision, you need to work out what the object is (ie is it a shadow) then you have to triangulate it. That requires continuous camera calibration, and is all that easy. If you have a depth "prior" ie, yes its real, yes its large and yes its going to collide, its much much more simple to use vision to work out what to do.
As far as distinguishing shadows on the road, that's what radar is for. Shadows on the road as seen by the vision system don't show up on radar as something the vehicle will run into.
The SAE autonomy scale is about dividing responsibility between the driver and the assistance system. The lowest revel represents full responsibility on the driver and the highest level represents full responsibility on the system.
If there is a geofenced transportation system like the Vegas loop and the cars can drive without a human driver, then that is a level 5 system. By the way, geofencing is not an "SAE level 5" requirement. Geofencing is a tool to make it easier to reach requirements by reducing the scope of what full autonomy represents.