upvote
I don't see how it is?

As said, the labels there are just randomly chosen things that made headlines, thrown on there to imply a clear causal relationship.

That's the level of "science" that gets you banned from campus. In an ideal world, anyway.

reply
Maybe a poor analogy but I see this as OpenAI releasing their GHz speeds or IPC metrics if we were to imagine they sold CPUs instead of AI services. It doesn't tell you much about the final performance but it does give an interesting core data point.
reply
I don't think that's the case, because with instructions per second, there is a clear relationship. Whereas here on one axis they plot GHz and on the other they plot matches of StarCraft won.

There is certainly some influence of the Ghz of your processor on how many matches of StarCraft one can win, but.. well. Bogus science.

reply
My point was that it is a core but not necessarily correlated metric and it sounds like you agree with that.
reply
Okay so, the graph is "informative" because it plots the perplexity value of the various OpenAI models over time?

That's it? That's all this submission has going for it?

Okay.

reply
That was all I'd apprehended of the post when I replied earlier.

The post argues that NTP perplexity is a good first approximation to model capability, and could therefore form the basis for an industry-wide commitment to pause model development at a given perplexity level until there's a consensus that AI alignment and safety has caught up enough that it's safe to continue developing even more capable models.

It's definitely not simply an ad.

reply
They're meaningful to anyone who understands the standard numerical objective used in next-token-prediction pretraining.
reply
I am so incredibly glad that this is going to be over soon, because the amount of weak replies like this is just.. insufferable.

The whole AI Bubble topic aside, this is just.. ugh. Intellectually repugnant.

There is just nothing here other than a war of religions. Nothing is to be learned here. There is zero curiosity. It's just a screaming match.

I hate this.

inb4 "no you"

Yeah man. Whatever.

reply
The graph shows the average uncertainty the models have when predicting the next token in natural-language text. That's a highly specific measurement, and informative anyone who understands it.

You're not under any obligation to understand it, of course, but if you ever get past this emotional reactivity, you'd benefit from making the attempt, IMO.

reply