As a proof of concept I've been working on a Jev style vision model that can rapidly answer boolean and choice questions based on images (See [0] but not the point of this comment). One of the reasons I wanted to make this concept is because I was also thinking about exactly what the parent comment suggests. I just haven't gotten around to getting (3d printing) a robot arm yet :)
reply