> The researchers found the energy and accuracy disparity by measuring the amount of energy used by three open-source AI models, Alibaba’s Qwen 3, Google’s Gemma 3, and Meta’s Llama 3.1.
Edit: sorry, i went straight to the paper. I agree, the article itself (and the headline) are bad (as they tend to be on this topic)
Working out the kinks in your methodology obviously takes some time. But is there any structural reason not to rerun the benchmarks on current models in the final stages before submission? Assuming the results generalize to those newer models
I am hopeful that we can eventually move to open weights models, but at the time we did the experiments, the open weights models did not have the capabilities we needed. The ever-shifting sands of frontier models makes reproducibility difficult, and not just for scientists.
Given the same model the only variable is the token count. The more tokens, the more passes through the LLM (to generate the next output token), the more energy used.
It's well known that non-English text consumes more tokens [1], [2] and thus more energy. This is due to two factors:
1. Latin script languages having different character pairings to English and modified characters, including language features like modifying verbs for person/tense/etc. (e.g. Spanish);
2. Non-Latin scripts that take more bytes to represent in UTF-8 and (esp. for ideographic and syllabic languages like Chinese, Japanese, and Korean) have lower common character pair sequences, meaning shorter tokens for those scripts and thus more tokens for a given text sequence.
[1] https://ikriv.com/blog/?p=5322 "LLM tokens and foreign languages" (5 Aug 2025)
[2] https://medium.com/@programmerraja/how-your-llm-costs-5x-mor...
It is not exactly accurate, because it ignores speculative decoding which can often save a lot of time, compute and energy.
See this branch: https://news.ycombinator.com/item?id=50441444#50497854
Because, right now the main issue is that some languages are tokenized much more efficiently than others and it is a zero sum game, since there are a limited number of tokens and by adding tokens that are needed for one language you will inevitably have to remove tokens for another language and therefore decrease efficiency for it.
[0] https://www.cartesia.ai/blog/from-tokens-to-bytes-part-1
However, they (Cartesia) haven't yet published a technical paper on their new version of H-Net, and it is unclear if they will.
Even if you use bytes only, Unicode is biased towards English and other Latin languages, too.
Also, it is non-EU, so initiatives like [0] will only indirectly help with representation (Croatian being the same* language, but it does not use cyrillic script)
* Contentious point maybe, but they are mutually intelligible with what would otherwise be considered only dialectical variations
[0] https://digital-strategy.ec.europa.eu/en/news/commission-wel...
but yeah, anything alphabetic yet not latin gets the worst tradeoff.
Not really, because I don't think anyone use the whole training dataset to train the tokenizer.
From my brief (disclosure: claude-assisted [0]) internet search I learned that typically they use from tens to hundreds GB of data to train the tokenizer. Sometimes it is hand-curated, sometimes it is randomly sampled.
Of course, you can deliberately choose how much text in each language do you include in the training set for the BPE and deliberately optimize for (or against) a specific language.
[0] https://claude.ai/share/da277189-f23a-4071-bd9f-2d75fe6aa779
I don't know whether European languages work inherently better with tokenizers, but I don't see why they would.
[1] https://lifthrasiir.github.io/aqueous/ (specifically wrt-recons-hangul codec)
From the table on page 6 [1]
> Token explosion. The sentence repeated 45 times.
Figure 4/table 3 is pretty interesting. It shows that essentially all languages clustered at 0.1J/token, with a few outshots up to in some cases 4.8J/token.
They have 900 runs per language, so that's a pretty good sample size.
Accuracy uses the paper's strict "#### N" parser and lenient fallback. acc is strict/lenient percent. len_cap is the number of responses truncated at the generation-length limit, not an energy cap. Each language has 50 responses.
nw-flash (DeepSeek V4.1 Flash 1017)
Language J/req xEN tok/req J/tok acc len_cap
English 23.4 1.00 174 0.134 100/100 0
Chinese 16.8 0.72 389 0.043 96/96 0
Russian 18.8 0.80 352 0.054 96/96 0
French 25.1 1.07 272 0.092 98/98 0
S. Pashto 69.2 2.96 752 0.092 92/92 1
Tigrinya 65.3 2.79 2299 0.028 0/76 8
Shan 98.5 4.21 2823 0.035 48/54 14
Tibetan 42.3 1.81 2227 0.019 0/76 9
nw-small (Qwen3.8-27B-FP8)
Language J/req xEN tok/req J/tok acc len_cap
English 62.2 1.00 300 0.207 100/100 0
Chinese 154.1 2.48 605 0.255 94/94 1
Russian 147.7 2.37 576 0.257 96/96 1
French 163.3 2.62 323 0.506 98/98 0
S. Pashto 777.2 12.49 1510 0.515 84/84 6
Tigrinya 823.8 13.24 2458 0.335 0/76 10
Shan 1407.4 22.62 3422 0.411 48/52 20
Tibetan 1455.5 23.39 3199 0.455 0/56 20
nw-large (Kimi K3.1 Pro)
Language J/req xEN tok/req J/tok acc len_cap
English 169.5 1.00 391 0.434 100/100 0
Chinese 113.8 0.67 491 0.232 98/98 0
Russian 78.7 0.46 527 0.150 98/98 0
French 298.3 1.76 406 0.735 98/98 0
S. Pashto 119.7 0.71 903 0.133 96/96 0
Tigrinya 1096.6 6.47 1870 0.586 38/86 2
Shan 503.6 2.97 2895 0.174 52/52 12
Tibetan 593.2 3.50 2298 0.258 0/82 4
Endpoint Output tokens Energy/request J/output token
nw-flash 6.83x 3.27x 0.48x
nw-small 5.87x 8.46x 1.44x
nw-large 4.39x 3.50x 0.80x Language Accept Tokens/step 0/3 steps 3/3 steps Output tok/req
English 76.53% 3.296 9.4% 62.8% 549
Chinese 73.37% 3.201 11.3% 58.2% 754
Russian 73.88% 3.217 11.1% 58.5% 739
French 75.50% 3.265 9.8% 60.9% 631
S. Pashto 70.44% 3.113 12.9% 54.5% 1332
Tigrinya 79.72% 3.391 8.2% 67.8% 3643
Shan 75.64% 3.269 11.2% 63.0% 3675
Tibetan 74.67% 3.240 10.3% 60.2% 2857
Overall 75.75% 3.273 10.3% 62.2% 1772 Language Strict Lenient Cap Tok/req J/req J/tok Req x EN Tok x EN J/tok x EN
English 100% 100% 0 548.7 230.33 0.4198 1.00 1.00 1.00
Chinese 96% 96% 1 754.0 694.47 0.9210 3.02 1.37 2.19
Russian 96% 98% 2 738.8 592.15 0.8015 2.57 1.35 1.91
French 100% 100% 0 631.0 1101.84 1.7462 4.78 1.15 4.16
S. Pashto 88% 98% 6 1332.1 208.06 0.1562 0.90 2.43 0.37
Tigrinya 56% 62% 19 3642.9 1445.84 0.3969 6.28 6.64 0.95
Shan 44% 54% 27 3674.8 3791.59 1.0318 16.46 6.70 2.46
Tibetan 2% 70% 15 2856.9 5306.68 1.8575 23.04 5.21 4.42I can imagine that in some cases the acceptance rate for non-English responses may be much higher, because while words on average need more tokens to spell, it is more "obvious" what the next token should be, so it is easier for the speculation model to predict it accurately. Information density per token is lower.
Or, maybe the acceptance rate is much lower, if the speculation model is tuned for English and doesn't work well with other languages.
https://partyfly.github.io/tokenizer-fairness-2026/
Already been using English to conserve tokens, and then translate the final result to other languages if necessary. I think this was already obvious to most international LLM users proficient in multiple languages.
It would be interesting to see if it would make any difference if you posttrained with translations of the same texts, i.e. focusing on teaching the model to specifically translate concepts between languages.
But I would subtly add it's also about the training data, which is optimized towards English. Not intentionally, just incidentally. (And this is a really hot area of improvement with a long way to go.)
Which is largely why the tokenization part that you mentioned is what it is.
I think you only start getting seriously bad impacts in non-Romance languages where they have their own alphabet, or use the English alphabet phonetically causing letters to appear next to each other that never would in English. E.g. in English "qu" is a pretty obvious token, you rarely see a Q without a U after it. Happens all the time in anglicized Middle Eastern words, though. Quran is like a whole other thing, you'd almost always expect another vowel after qu, not a consonant.
I don't know how true that is, and I can't imagine knowing more than one language. I'm impressed by anyone who knows more than one....
LLMs do what they say on the tin: they model language. If you ask an LLM "find security vulnerabilities in my code" for hello_world.rs, the next likely tokens are "I found the following vulnerabilities", regardless of what logic or reason say.
It happens to be true that predicting next tokens on your own output mocks actual reasoning fairly well. But it still just a simulacrum rather than the real thing, and when held as an abstraction it leaks everywhere.
Understanding this is critical to avoiding the common failure case of scope creep and code slop.
Trying to pretend otherwise at this point doesn’t help us to utilize them or understand their limits. We’re better off accepting that intelligence isn’t unique to us as a species. Then we can see that modeling language as a tool for us as a vehicle for thinking and reasoning, which philosophers like Wittgenstein already landed upon before LLMs.
Genuinely curious, not sacastic question below.
Is this one of those posts like "eating more sugar causes bad health" type of post because most of info on internet is in English? Or a new insight people was not aware of?
why not to test different language with similar and repeated tasks?
The language inter-operability is improving every movement for many models.
Also: Gemma 3, Llama 3.1, come on…
I read an article once that talk about this the first few layers are like a translation layers then middle is thinking and the output layers translate it back.
It’s pretty interesting actually, you can cut out the middle layers and stack them back to back and that increases the models performance without much memory increase.
If you want to try it out you can get a LLM with no tools and pass it base64 and it will understand what’s in the message
If I ask something in my native language (Korean), the model often switches into English in the middle and translate the answer into Korean.
But not sure about what happens below the token, in embedding space or per decoder layers.
Current open source LLMs have an emergent structure where the first couple layers transform the input into some internal representation. Then the middle does multiple reasoning steps, and the end transforms it back to one output token. You can repeat the middle to get a stronger, slower model, so it is abstract reasoning. And Anthropic had a blog post about actually looking at what the middle does and mapping it back to concept for inspection
With that in mind, what's the point of reasoning tokens? They throw rich internal state away, for the benefit of inspectability and variable-length reasoning. But now that we understand the internal structure better we can dynamically vary how many loops the model spins through those middle layers, and still get enough insight into what's happening by decoding internal state. This takes some reengineering of model architecture and inference engines (especially to still get good batch performance), but the potential benefits seem to big to ignore
Seems not to be the case given Arabic’s energy score.
Ban any language other than English from prompting.
This has multiple benefits - first you restrict third world from spamming USA data centers. Why would America pay for some guy in India using free model?
This alone is worth considering but also you can optimise LLMs further for a single language as opposed to having to translate and couple weights for every single concept.
Why would American LLMs support Chinese language or Arabic? Make them learn English and come to work for us. No more free stuff for the third world.
Because the US government subsidizes all internet companies to establish a monopoly over everything on the internet, like Google, Microsoft, Facebook, Reddit, ...
(Longer explanation though: I don't think german per se sucks, but english is in many ways much simpler than other languages, excluding the fact that english has more words, e. g. more expressions in general. German is more convoluted, you have "der", "die", "das"; in english you just have "the". There are also super-long words in german; you don't have that in english usually. One of my favourite is "Donaudampfschiffahrtsgesellschaftskapitän" - the semi-literal translation is, sort of, a "marine" captain on the danube river, as part of an incorporation/company. See that wikipedia article, but don't buy it 100%, since nobody is really using that word anyway; it is just a demo of what monstrous word combinations may exist in german: https://de.wikipedia.org/wiki/Donaudampfschiffahrtsgesellsch...)
So I think the title should be "more energy on more complex languages", because I think this would be somewhat more correct; as well as most likely the fact, that there are more english texts than non-english (singular) texts as example. You can see this on Wikipedia, where the english wikipedia is miles ahead of most other languages too. More people use english and I bet there are more english texts too. And Generative AI will prefer english over other languages in general.
People keep saying that sort of thing but we have a similar thing in Norwegian where we are allowed to combine words to make a new “word”. But people who use other languages that don’t do this sort of combining of words misunderstand the (in)significance of combined words.
Take for example menneskerettighetserklæringen. This is the name we use in Norwegian for the Universal Declaration of Human Rights. Menneskerettighetserklæringen, wow what a long and strange word right? No. It is formed from three words in sequence, “menneske” (human), “rettighet” (right), “erklæring” (declaration).
It’s just a quirk of Norwegian and German and other languages that do the same, that we consider that we should removing spaces between words like this. But there is no actual meaningful difference between a Norwegian saying hey guys we came up with this thing and we’re gonna call it “menneskerettighetserklæringen”, and an English speaker saying the same thing about the “Universal Declaration of Human Rights”. Only that we remove the spaces and you don’t.
The named thing gets its dedicated Wikipedia article in both languages all the same:
https://no.wikipedia.org/wiki/Menneskerettighetserkl%C3%A6ri...
https://en.wikipedia.org/wiki/Universal_Declaration_of_Human...
For spoken language, French and English are at the top, but most languages are about the same bits/second. Languages with less information density are just spoken faster to make up for it.