As a client, you're still paying for the compute even if it happens on the service provider's side. Running the same computation twice (perhaps by another provider) is 2x the cost; but if you ask the provider to make this proof, that increases the cost 70-95 times according to the article.
replyWith that level of overhead, all kinds of otherwise inefficient approaches of getting compute start to seem comparably cheap. This is progress, but it still needs like a hundredfold improvement in order to become practical; I don't see a usecase where the benefits of this proof would be worth 70x or even 7x increase in cost.
Since it's CPU vs GPU, it's not apples-to-apples. But in terms of wall-clock time, one inference pass is surely faster than 17 seconds. Plus there's the proof generation step, which the post describes as ~100x slower than generating the response itself.
reply21s on a single core for 1024 tokens, that’s ~50t/s to verify and a 6.5 meg download. I’m pretty sure you won’t hit that for a 400B model at 4 bits, but then maybe you would I don’t track that performance so much. You’d need an enormous amount of ram to handle it though.
replyYes, the generation is slower but I think here the verification itself is more efficient than regenerating the output.
The bigger factor is that (IIUC) ZKP-based verification is constant-time in the size of the output, so it definitely beats regen-based verification (as long as the constant is reasonable). And the ability to verify on consumer hardware instead of a beefy GPU is important too.
replyI think there is a place for both. Deterministic inference has other benefits, like "compressing" an output into input+model, as well as more esoteric applications like solving coordination problems. But if we get fast ZKPs for real frontier models, I'll be overjoyed!