You can do that with smaller models at home. Gemma-4-E4B will run on a 12gb GPU, and supports audio, image, video input
reply12GB GPU is a lot
replyjust use one of the various cheap gemini models
replyIndeed, Gemini really is incredible at image analysis. Yesterday I pointed it at some sloppy handwritten notes and asked it to add up the numbers in the right column, and it did it no problem. I've also used it to find out what TV show or actor is on screen, and various other things. It's quite impressive.
replyGemini pretty clearly has the best underlying model, and the worst RL and post-training of the lot.
replygemini models are also fantastic at understanding non spoken sounds
replyOr you could just use a CNN...
replyCNNs are not SoTA anymore when it comes to large models, and also are not used to provide interpretations of images as text, but rather to classify, do semantic segmentation, etc.
replyCNNs are fine when trained with a good recipe. There are very few good studies comparing them with proper hyperparam search and all the training tricks applied consistently. Transformers are good but ViT vs CNN is not some settled issue. Transformers are more hyped and more popular with the tech enthusiasts who just read forums and news, but if you need stuff done, CNNs are still great.
replyI agree, but since we're talking about imagine understanding with text output, clearly a CNN is unsuitable. My previous comment was overly reductive and CNNs can still be SoTA depending on your performance metrics. I spent the earlier part of my career training CNNs, and they are very pleasant to work with.
replyCan you say more about that? I haven't kept up.
replyCNNs excel in vision tasks where you have limited compute, limited memory, limited data, and want something that works super well and quick. People usually don't hook CNNs up to a transformer to get language understanding either, you have to train bespoke CNNs for specific tasks
replyViTs excel where you're unbounded in compute + data and also want text understanding or have a conversation about an image
Transformers are superior
replyWhich?
reply