It feels like the real cost might be negative though.. They use frontier math as a way to test improvements in their model. So solving the problems is like a positive externality, but the important thing is they can verify that the model is improving instead of looking at useless benchmarks. Plus it is good for marketing and attracting talent.

I didn't argue that knowing the total cost is uninteresting. What I was saying is that realistically the total cost is:

Hours needed for prompt + Hours needed to check result + API costs.

You don't say "well let's add together the total yearly compensation of all the engineers and mathematicians at OpenAI that were involved" and throw that into the total cost. That's simply nonsense accounting.

The actual comparison you are making is some university researcher weighing between getting a grad student (several tens of thousands of dollars) vs typing up a prompt and sending a request to OpenAI for inference (as mentioned in the article, around $2000 in API and maybe a few hours for the prompt and harness).

[dead]