> FYI there are no FOSS LLMs

FYI there is and been for a long time. Won't claim they're SOTA, but they exists. From the top of my head, I think Olmo (https://allenai.org/olmo) was pretty early, but been more since then too.

I agree most releases today that claim to be "open source" actually aren't, but that doesn't mean "FOSS LLMs" don't exists at all.

I believe Nemotron also publishes their dataset.