If you’re using the latest models( Opus/Fable/Astra/ maybe Sol) and the default harness, along with a little guidance against the worst tendencies like over-writing tests, you really shouldn’t be getting the wrong things built. Like a coworker, you can’t throw it blind into a monster codebase without some guidance, but otherwise the latest models genuinely just work TM.
replyTesting, testing, and more testing. In the project I'm working on right now, about 2/3 of the code is in one way or another related to testing. As a human coder, I'd never spend that much time on building test harnesses, automating regressions, writing unit tests, etc, but it's cheap when you can have a robot do it.
replySo you assume code written by the agent canno be trusted? But how comes you are trusting the tests?
replyAt my work we still have a QA dept that runs the manual test suites before each release. We have also automated almost all of those manual tests and they gate every merge to mainline.
replyThings are still found but normally it is a new test case or an edge condition we don’t have an automated test for. The ever growing e2e suite has reduced the amount of regressions pretty drastically.
EDIT: I forgot to note that automated e2e tests were always the answer to improving quality. AI has just made it cheap to have several layers of test coverage.
To start, you make it a rule that every test must pass with the fix being tested, and it must fail if the fix is missing.
replyYou of course use another AI to test your tests, have you been living under a rock? /s
replyHow are these /s posts even allowed? You contributed nothing to the conversation other than noise.
replythanks for the insight throwaway
reply