undefined

points

[-]

That's an example of why it would be useful for someone to actually do it. A random commenter on HN is one thing. A direct comparison on a brand new app that isn't part of any training is another

by CaveTech16 hours ago|

parent|

[-]

I’m highly confident that prior exposure is irrelevant at this point. I work on vulnerability detection at a hyperscaler.

by HDBaseT15 hours ago|

parent|

[-]

That's an example of why it would be useful for someone to actually do it. A random commenter on HN is one thing. A direct comparison on a brand new app that isn't part of any training is another

by GuB-421 hours ago|

prev|

[-]

Before Mythos is released to the world at large and not just to select people behind NDAs, I will treat it as its name suggests: as fiction.

Maybe it is the real deal, but in a world of overpromising and underdelivering, I prefer to be skeptical.

by enraged_camel12 hours ago|

prev|

[-]

People need to stop repeating this because it’s not true. Yes, other models can find the same vulnerabilities Mythos found… if pointed at the exact code that has each vulnerability. It does not mean they are nearly as capable when starting from scratch, or when chaining multiple (often very obscure) vulnerabilities).

by adrian_b5 hours ago|

parent|

[-]

Anthropic themselves have explained that the harness for Mythos has a very important role in finding the vulnerabilities, because the model does not start from scratch, but the harness runs the model many times on each file of the code base, with different prompts, where the prompts evolve depending on the results of the previous runs.

First with more generic prompts, to determine whether it is worthwhile to do a detailed analysis of that file, then with more specific prompts to identify the bugs, and eventually with a prompt that requests a confirmation that a given bug/vulnerability exists.

For a proper comparison between some other model and Mythos, you also need such a complex harness. If you just tell to an LLM "find the bugs", and it does not find a vulnerability known to have been found by Mythos, that is a totally invalid comparison.

The final results provided by Mythos, like a PoC exploit or a patch, are also generated with a prompt that points to the exact code that has the vulnerability (which is supposed to exist based on the results of the previous runs).

by loeg4 hours ago|

parent|

[-]

My take from the SCW interview is that the Mythos harness isn't all that important and the author thought it would be even less important with future models. But maybe I misremember.

by bitexploder4 hours ago|

parent|

[-]

Anthropic has a vested interest in downplaying the harness relevance. In my experience harness really matters. More capable models are great, but current models are enough if you put some engineering effort into the harness.