upvote
> As I work in english and use them in english, I wonder if using LLMs in a different language to code renders a different result as well. Like, if some of these benchmarks were made in other languages, would the result be similar.

I am not a Chinese speaker but my understanding is that all of the models have substantially different behavior in Chinese, to the degree that it's kind of like a second model. Would be interested in hearing more if anyone has direct experience.

reply
Fwiw, it's been very easy to nudge Qwen 27B/35B to think in Chinese. No need for <think> prefills like "思考:" ("Think:") or such.

Relatedly, when chatting with frontier models about science education content design, I've found it very helpful to mix in Chinese education terms. In English, for example, NGSS is such a massive attractor, discussing nearby topics often yields NGSS "slop". And "estimation" (educational) in the US means one (dysfunctional:) thing, which similarly distracts. Perhaps if AIs become increasingly multilingual, but remain weak at deep conceptual reasoning, it may be fruitful to have multilingual thesauruses, to use language-associated cultural conceptual differences as a way to convey conceptual nuances with which LLMs otherwise struggle?

reply
This is an interesting post, but as a non-American it's hard to understand what's it's alluding to in some parts.

The one criticism of NGSS I found from a quick skim of https://en.wikipedia.org/wiki/Next_Generation_Science_Standa... is that it teaches evolution.

What does "estimation" mean in the US?!

reply
The newly opened possibility to learn“how non-English cultures do X” is criminally underutilized.
reply
> In English, for example, NGSS is such a massive attractor, discussing nearby topics often yields NGSS "slop".

For that particular attractor, even German or Spanish or so might help avoid it? Or perhaps even just using British English?

reply