Generating datasets to train more efficient models is a common use case for VLMs, especially frontier ones. It makes it much cheaper to create that initial dataset and you can abuse the nondeterminism of LLMs to identify data for human review (if they don’t converge, escalate to a human).
replyEspecially the pill counting example. The best model was shown at 81.1% accuracy, which is a terrible rate for pharmacy scenarios. It seems like implementors would be better off instructing the models to use deterministic tools (like OpenCV) until the models are at 99.99% accuracy (or whatever an acceptable error rate is for pharmacy techs).
replyI think that is because people perceive OpenCV as 'hard to use' and LLMs as easy to use.
replyOpenCV is no longer hard to use, it just takes longer. Still, a little more complicated than asking LLM to count.
replyTo use an LLM, you just prompt it with an image + text saying "count the pills in this image".
To use OpenCV, ... you just prompt an LLM with an image + text saying "count the pills in this image, using OpenCV instead of eyeballing it".
(I like to throw in "produce intermediary artifacts so I can see the process" for more difficult tasks; this helps the model avoiding making hallucination-prone leaps and gives more opportunities to self-correct. At a cost of extra time and tokens, of course.)
Using OpenCV without an LLM? Nah, not touching that, I don't have free weekends to waste anymore.
I no longer use it but never felt it was particularly complicated, but since the days of resnet there are much faster ways to the goal.
reply