upvote
What will you do when the prompt was "Figure out the bug and write a report for me". Not saying it was in this particular case, but I think at least in other cases, it will be.
reply
Prompt an AI to figure out the bug and write a report, of course. If that is actually useful.
reply
"it will be"

Trying to figure out what to do based on possible future scenarios has a place, but not here when we are talking about a concrete present problem.

reply
> What will you do when the prompt was "Figure out the bug and write a report for me".

The rational choice would be to cut your losses and stop reading at that point. Once you realize zero effort went into the prompt, there's no longer any reason to read the output. The age old truism still applies: garbage in, garbage out.

If you think it's worthwhile, close the issue with a comment: "please rework this and show your work next time". Otherwise just close it without commenting and move on.

reply
It didn't figure out the bug: the report is largely nonsense. It merely did a bad impression of having figured out the bug, by stringing together several observations into something superficially resembling a narrative. Providing those observations without the gibberish framing might be useful.
reply
The parent's point still stands. In many cases it will have figured out the bug. In many cases it will have produced a small, clean, deterministic reproducer.

In my daily work I see these cases. It does help that the bugs that are filed contain a test case and some analysis by the agent.

It would not help to get ten identical bug reports all saying "I asked my agent to find a bug by prompting it with "find a bug and produce a test case". It found a nasty bug and a really nice reproducer. I'm not including its output here. Good luck!"

reply
That I agree with (not instead, in addition)
reply