That's a great answer, thanks. I'd missed that detail.
replyHonestly, a lot of those issues could be remedied with more optimized kernels, which LLMs could definitely write quite well now. I wouldn't be surprised if some deployments of Gemini Flash switched to diffusion for cases where there is low utilization, since Google showed you don't need to retrain an autoregressive model to rework it for diffusion inference.
reply> since Google showed you don't need to retrain an autoregressive model to rework it for diffusion inference
replyIIRC you do need to retrain the model, just closer to the level of finetuning rather than e.g. pretraining from scratch. Which also means that the model will perform differently with a different capability profile.
If that's true, that's the real answer, or more like anty-answer because you're saying basically: hey, nothing should stop them.
replyGoogle's gonna Google...
replyI wonder if it'd work better as a local model
reply