I work in this field and this is 100 percent true. It's really hard to learn about too. A lot of textbooks go over the algorithms in the chips in an idealized form. The actual versions are so messy and different that the textbooks aren't even useful sometimes, especially if you work on custom ISPs. It's cursed, but it's fun.
Unfortunately no. It's all proprietary and locked up. However, I can say that the optimizations are not always about IQ (though that is a major factor). It's also about making things run fast enough in a low latency environment. Those two requirements lead to strange hardware designs that lead to strange register interfaces.
Wait is IPU6 working now? My work laptop has it and the only thing I know is it might work provided I use some user space relay driver and a bunch of other things.
I looked and looked and looked. I don't know why but IPU7 is reported to be pretty good. But IPU6 might still be in perpetual limbo; I don't know. What kernel are you on btw? Sorry for your pain.