We split them. The agent writing the code never writes the checks it's judged by, and those separate checks caught way more than its own tests did.
replyNot sure about that. This was structurally the same before AI: the author of the code wrote the tests. The tests still helped.
replyThe author of the code was physically touching the internals and potentially intuiting obvious shortcomings as they worked on it. That very crucial part is now gone.
replyI like fault injection (fuzzing, chaos), and mutation testing. I think introducing these helps the AI write better tests and helps having some confidence in the tests that are written.
replyYou should be using two (or even three) different LLMs. For example: Claude to do the coding, and Codex to do an adversarial review. This can go back and forth for several iterations before converging on an acceptably "correct" version of the code...
replyOccasionally, Claude may not be able to fix the issue – and it may turn out that Codex can, because it has a different approach.
Either way, at least two LLMs is the only way to get (mostly) reliable code.
You can automate this entire orchestration with zeroshot.sh, mentioning it since it is relevant here.
replyClaude, ChatGPT, Google Gemini, then Replit...
reply[flagged]
reply[flagged]
reply