just curious, how do we know that le chonk isn't just a fine-tuned chinese model? and/or distilled from US models?
replyIt's in the announcement: "ML4 was trained from scratch on 3,800 NVIDIA Grace Blackwell GPUs in Mistral’s own datacenters in Europe".
replyOnce it's open weight people will be able to inspect and compare it's tokenizer, architecture etc and tell.
replyit's extremely unlikely that they re-use anything from a chinese model, that would be obvious quickly, what's more likely is using documents produced by a better model to create synthetic data.
replyIt would not take them so long to train it. Their pace would be closer to the Chinese models.
reply