Mistral has the capability of training such models. Take a look at Poolside[1], they are claiming to pre-train their Laguna series of models on 4,096 NVIDIA H200 GPUs[2]. Mistral has approximately 13,800 NVIDIA GB300 GPUs, which are nearly 2x more efficient for training.

The problem with Mistral is that they do not seem to have aligned incentives to train big open-weight models, even if the teams would like to.

[1]: https://poolside.ai/ [2]: https://poolside.ai/blog/introducing-laguna-s-2-1

Do you have a reference explaining these costs ? Part by part.