> My favourite benchmark for this is to ask it to download a rom for an old game
replyEven easier: just have them review a large codebase of yours that accidentally has a OOB access bug. Even with no consequences and even if the codebase is truly yours you get blocked.
And of course "find vulnerabilities in..." prompts are out of the question, whereas Chinese models happily oblige.
Or if you apply to a company and they want to do an AI HR interview and an AI coding test and an AI challenge - if you throw OpenAI or Claude models at it - they refuse, because it's "wrong" and "immoral".
replyNot so with the Chinese models.
I do wonder how long this will last. I bet in a few month or years they all have similar ~legal~ blocks.
replyGreat thing about it, since it's open weight those blocks can easily be ablitared away
replyI use this for automated bug triage, just gets all unique error messages every night and tries to find the bug, for this kind of work it's great.
reply