upvote
They already are?

There’s a measurable performance tradeoff versus gqa so there’s reluctance.

For the most part though the new deepseek v4 tech is hca and mhc and people are still catching on like with moe and rl. Wait for 6 12 months, minimum time for next pre train.

reply
Are they not?

The big US labs are opaque and don't publish much of any technical details anymore. We don't know what they are or aren't doing, honestly.

reply