Drivers can also be partially implemented / buggy depending on the device. (And I'm not talking about closed source stuff, even things like usb or touch screen drivers).
I would say that the major blocker is that nobody really cares about these devices and the task itself to mainline them is gigantic.
Most SOC have 3 or maybe 4 contributors maximum and the phonedev kernel mailing list gets a new patch only every 2 days. The whole phone mainlining community is probably less than 60 people.