upvote
I don't think these results show that.

I think they show that additional prompting for testing approaches have wide differences in error rate (worst has twice the error rate of the best) but actually no extra prompting is pretty fine and most custom prompts are worse than no prompt.

reply
Most developers are not good at testing. That's why the role of test engineer exists.

I've seen in multiple projects things like assertTrue(true).

I'm sure the agent is better in testing than average enterprise developer.

reply
big tech is moving / has moved away from the role.
reply
I thought so too but it's probably because I never review the tests. I made an exception recently and found tons of

    assert(CONSTANT_CONFIG == valueOfConfig)
or tests for keywords in prompts:

    assert(prompt.includes("repo url"))
reply