> we’ve seen huge lifts in all of these areas, not just the verifiable ones.
replymost gains are still coming from data. isnt that supposed to 'run out' though?
Labs spend billions hiring experts to generate new data, and better models can better filter existing training data and generate new synthetic data. There’s no reason for that to run out, it’s just expensive.
replyYou could view this as just continually patching a leaky ship. But it seems to work.
That is because there is human annotated data there. Every session you or I used, then of course paid human feedback on repos (such as the recently famous example of meta forcing their employees to).
replyThis is _much better_ data than 1/0 verification, it is as good as a gradient.
Automatically verifiable tasks improve faster since well, its automated.