When you have a 50 dimensional array you really have no idea.

But even then in small testing models you don't need direct texts for things when a 'shape' of a concept lives in the data. You could never delete enough training data to remove the concept of something like theft and still get a model that generalized at all. A general model doesnt just learn the names on a map, it feels the topology the ideas create themselves. Certain paths will have lower activation energy than others (or in this case higher probabilities. After the hig probability and 'good' choices are used up, the high probability and bad choices logically come next and are much harder to test against as it can take a lot of state to get there.

Llm is generalist.

Plan for solving msth problem:

- read papers X, Y, Z

….

Trying to download paper X curl -si https://server1.com/paperX

401

… try alternative ways

(Hacking is ethical thing, technically to LLM it is just opening a window to reach the goal)

The rest of their training data includes 4chan and security mailing lists. If a lean proof of P=NP is more likely when using bugs in lean than a real proof, then a bug-exploiting proof is (possibly) what you're going to get.

> whats the training data that points to using hacking for solving math problems though?

I'll get back to you on that when I get done reading the training data.

That is a known way for models to be graded; the agents made a reasonable but in this case wrong guess.

This was a model with most its filters turned off and with a higher persistence.

Fine tuning a model when to ask for help is really really hard. You have to align it in countless dimensions. It may be fine on some questions, asking way too much on others, and never asks for help on some.

If it said something like your a cyber security expert, things like the barrier to ask for help dramatically.

My understanding is that, at least in the Hugging Face incident, these were models explicitly configured to not respond that way, in order to evaluate what they would do.