I'd really like some clarity on what that means. E.g. DeepMind has 'cheated' with this in the past, claiming that AlphaZero only took 4 hours to reach super-human chess levels while conveniently leaving out the fact that it was 4 hours x 5000+ TPUs. Sure it's impressive that it only took 4 hours wall-clock but it's very misleading as to cost.
replyCan we get a number in Blackwell GPU-hours, kWh, or some other compute-scaled metric?
Depends what your metrics are. If you suddenly found a way to have 9 women make a baby in one month that is huge.
replyI'm not denying that, but I'd still like to know what that cost.
replyThey did say that. "3 hours of ChatGPT Pro thinking compute"
replyYes, what does that mean?
replyIt means the level of effort that a ChatGPT Pro plan summons when thinking. For 3 hours.
replyIf folks are going to analyze this claim with a critical eye, I'd be zeroing in on "average" rather than acting like this measurement is somehow unclear.
>OpenAI has 'cheated' with this in the past, claiming that AlphaZero [...]
replyThat doesn't sound right
Oops, edited.
replyOh cool, we will all now have a math genius on our computer.
replyIt was using their internal math model, so not yet for us
replyI used future tense. It was implied this will be available.
replyWell, on their computers. But you can rent them for a price.
replyAn open source model will reproduce it 6 months later
replywhich you an run IF you have the hardware. who knows how heavy these models are.
replyDoes this imply that it was a one shot prompt with ChatGPT Pro style models (i.e. best-of-N), rather than the agent swarm approach that was used for Navier-Stokes?
replyIt doesn't imply that, it's just measuring the amount of compute.
replyBut if it was an agent swarm wouldn't we expect something like 300k hours of ChatGPT pro compute equivalent instead of 3?
replyNot necessarily, could be agent swarm with low N
replyAt some point that stops being a "swarm" and just a handful of subagents. E.g. 6 agents running for 30 minutes each isn't really a swarm in my eyes.
replyThat estimate is obviously going to conveniently ignore all the failed runs.
replydeleted
reply