> Images are converted into tokens based on their dimensions, and these tokens are billed together with your text tokens.
> Before inference, every image is automatically resized:
> - Images with a total pixel count below roughly 384×384 are scaled up while preserving their aspect ratio.
> - Larger images are scaled down while preserving their aspect ratio so that the total pixel count after resizing is roughly that of an 800×800 image.
> As a result, there is an upper bound of 384 tokens per image: for example, a 2000×2000 image and a 5000×5000 image consume the same number of tokens after resizing. When a request contains multiple images, each image is counted independently under the same rule—there is no separate calculation for multi-image requests.
400 tokens per image results in 2,500 images per dollar, if I’m not mistaken.
edit: format.
The really wild one is even blind models will do this and they'll try to run stats on the pixels to figure out what it looks like... the even wilder thing is that it kind of works!
I don't think the n by n subgrid fixes this the way most harnesses do, as it'll fail to count things if you have more overlap and fail relatiomships if you have less
Maybe there are some use cases where you need high detail everywhere at once, but for OCR of small text and the like a zoom ability should be sufficient
Can't remember if I stole this idea from some existing public harness though, can't remember. If someone knows of public harnesses that do this already, please share them :)
I asked it "what time does the clock show?" (both on reasoning: high)
DS answered: The clock shows *5:10* (and 45 seconds). Here is the breakdown: * *Hour hand (red, shortest):* Pointing at the *5*. * *Minute hand (green, longest):* Pointing at the *2*, which represents 10 minutes. * *Second hand (blue, medium):* Pointing at the *9*, which represents 45 seconds.
Qwen answered: The clock shows *8:10* (with the red second hand on the 5, i.e. *8:10:25*).
- *Hour hand* (short, blue) → 8 - *Minute hand* (long, green) → 2 (10 minutes) - *Second hand* (thin, red) → 5 (25 seconds)
Correct answer is 08:09:25.
Snark aside, I’m not sure that these gotcha tests are any more useful than asking politicians gotcha questions. Sure, the model can’t tell me what time it is, but it can code the Wang algorithm for noisy audio matching in one shot. Maybe this is just me being an optimist, but this is my hiring philosophy and I guess maybe now my llm philosophy: I’m not interested in seeing how dumb I can make you look, I’m more interested in how smart you can be.
- A superintelligence that will usher in an age of human enlightenment
- A superintelligence that will usher in an age of human enslavement
- A really cool way to rake in trillion of rich VC/investor money by promising you're building a superintelligence that will usher in an age of human en[slave/lighten]ment
- A transformer model for predicting output tokens given a series of input tokens, informed primarily by reddit, stack overflow, and 6000 years of classical literature.
- A replacement for white collar labor. Start now or join the permanent underclass.
- A convenient fuzzy-find tool also capable of some probably-correct code generation.
- The ultimate customizable text RPG experience (you can pick if G stand for game or...)
And so on.
So, some people see a new model and check for how close humanity is to enslavement. Some people check to see if it got better at fixing broken unit tests.
The whole point of LLM/FMs vs good old fashioned ML is generalization to unknown domains, not just unknown tasks. The hunt for "gotchas" is the hunt for "not in your training data".
> Sure, the model can’t tell me what time it is, but it can code the Wang algorithm for noisy audio matching in one shot
This is about a _vision_ model.
Also worth noting that both models got it wrong. Qwen made a mistake that humans very good at reading clocks would make. Deepseek made a mistake that a human who had just learned to read clocks would make.
I'd recommend non-thinking for any non-prompt input, and leave the thinking where it has to actually reason.
and qwen still answers: "The clock shows *8:10* (with the second hand on the 5, i.e., 25 seconds). - *Hour hand* points to the 8 - *Minute hand* points to the 2 (= 10 minutes) - *Second hand* points to the 5 (= 25 seconds) So the time is *8:10:25*, or simply *8:10*."
Are we more forgiving because it’s the same type of mistake a human would make?
"The professional failed its task!" // "Laymen would have failed it too".
Which makes no sense.
I can't actually remember where I learned to read a clock, it might have actually been in school. I guess that means they don't teach it anymore. (Everyone's phone shows the time anyway...)
And then I realized, wait a second... you're testing the harness not only against a difficult benchmarking problem, but it's one you're literally never going to use the coding harness for either, lol. I don't write programs that read or interact with sheet music and I never will.
tl;dr Being frustrated that a "state of the art" vision model doesn't have perfect vision is a fools errand.
It can read and extract information from screenshots and PDFs just fine (my setup). No need to worry about edge cases.
Anecdotally, I had to tell 0731 to refrain from viewing screenshots since it kept breaking its sessions by trying to read images.
It's expecting you to have done at least something besides select DS4 on Ollama, essentially.
It's useful but for OCR and a lot of other applications it needs to be a bit higher (eg: putting in a full A4 / Letter sized page)
I am using a stripped-down minimal version of it which I uploaded here, since I am not a fan of huge dependency trees: https://github.com/99991/simple-pp-doclayoutv3
Another recent model for this task is Unlimited-OCR: https://github.com/baidu/Unlimited-OCR
Interestingly, v4-flash performed several points worse on DeepSWE at 53% +/- 4%. Assuming this result is verified by DeepSWE officially, it would mark a significant advance in Pareto cost/performance on software engineering tasks.
Still an advance, I just thought it worthy to note Sol isn't nearly as impressive on the cost/performance frontier as discounted Luna.
This is useful for a reasonable amount of use-cases, but I think the watershed rez will be around triple that, ~1080p, which is enough for almost anything, except small text and subtle details.
Nevertheless, as a component, we will undoubtedly implement multimodal support — and we are already doing so. We plan to develop relevant models, ensuring that versions like V4 and subsequent iterations will natively support multimodal functionality.
Achieving excellence in AI training does not require a global model or even multimodal approaches—by narrowing the scope of AI training and eliminating multimodality, certain tasks may remain unachievable without compromising the algorithm's validity.
Multimodal approaches ultimately need to be implemented.
I really missed this feature when I had DeepSeek code a small game for fun. When writing UI and rendering code it could execute the game and get screenshots back, but then had to rely on my feedback on what had gone wrong. Models with vision can do much better here, finding more issues on their own
I've not used it myself, but it's there.
Also used it for 3d printer control once, had it diagnosing issues, calibrating my Tradrack MMU and canceling failed prints autonomously from a couple of cameras placed around the printer.
It makes running much, much longer feedback loops possible. Although you can mix and match non-vision and vision models simply by invoking a vision model when you need one, as I like to use non-vision models like glm-5.3.
1. process graphs and charts
2. process handwritten math formula, also chinese characters writings
3. process design sketch and wireframe
4. process scanned documents
... etc
in fact these transformer models currently suck for surveillance, too slow and expensive. There are already faster and better facial/gait/object recognition models out there.
this is such good way to learn something for me.
Edit: I see it has limited resolution. Luckily I just built a vision worker plugin for DSH that routes image input to Kimi K2.6 on Cloudflare.
But if not, does anybody know a recommended way to attach vision to deepseek flash (on a self-hosted infrastructure)?