It had a very non-standard RSA key implementation that was obfuscated heavily. As well, it has an online license check at first run, and that part typically trips up most of the local models I've tried. I've been running this test for about a year now with different models, and it was the first I've seen not only figure out the RSA key implementation, but the first that didn't just give up once it saw the online license check. Even though it's only a first-time launch check.
That's why I call it one of the hardest, because in my experience, it has been. It's the first local model I've seen pull it off end-to-end. For some of the reverse engineering work that I've done with LLMs, none have been as consistent as this particular test at highlighting a model's failure in this domain.
I have access to Daybreak Blue and I'm approved for Anthropic's Cybersecurity program, so I might run the same test with both of those just to see, because it's been a while since I used a frontier model on this test. I imagine they'll make relatively light work of it, though, assuming it doesn't trip the relaxed guardrails.
What quant and what abliteration of qwen 3.8 27b has worked for you? I'm concerned that too much quantization and it can't do the work anymore.
Agents (even ones powered by small models) do reasonably well when provided an oracle to work against.
You mean when the cocaine piracy parrot has something to plagiarise?
It's fairly successful but hits the time constrains and reduces the "value" of getting a local model to develop software.
It's still a bump in productivity.
if your tests and docs are possibly incorrect, and you're not writing the code.. how do you know if it even works? for extremely simple software you can just use it but for anything with access to disk or the network or with user options...
you sound psychotic. actually. so nevermind, LLM psychosis is extremely common on this website, that's def all that's happening here
In local coding, the screen scrolls enough to actually read it.
But that's fine. enjoy your misunderstanding.
It is probably no coincidence that AI is exceedingly good at finding small counter examples. But for the Riemann hypothesis no such counter examples exist. And likely none exist.
Counterpoint: P vs. NP.