> to conduct effective evals, you sometimes need the model to have (restricted) access to the internet

That's the thing. These evals were not effective as they failed to evaluate if the model was hacking or not.

Further shy is access to the internet required at all during evaluation? They could run the evals in a fully air-gapped system that simulates the internet. While that's expensive its not unfeasible.

> you sometimes need the model to have (restricted) access to the internet.

Poor excuse. Internet is prod env, for testing do a proper test env.

Just give the models in training access to a snapshot from archive.org - which frontier labs surely have a crawled copy of already.

If they aren't competent to run the evals safely, then they shouldn't run them.

The huggingface incident seems to have resulted from them partially mirroring a repository, but for no reason I can see not fully mirroring it, after which they could have unplugged the ethernet cable. This is just stupid.

Yes, they're stupid. Maybe also highly intelligent. But stupid.

Also irresponsible.

[dead]