I've totally had latest models just say "here are minor nits but this is good to ship" when reviewing code. Claude is a bit more verbose usually but Codex by default is pretty terse. My experience anyways. They also take pushback on suggestions (this isn't actually a problem because X) and they'll agree and say ship it (hopefully only when you're right and it's not a problem ;) ).

I personally rely on this behavior of LLMs. For example, for reviewing prose, I might ask the LLM to find five issues. If there are obvious issues I missed, then it'll find them. But if all the issues it lists are minor or hallucinated, then I can have some confidence there weren't any glaring issues.

I do the same for LLM code review comments: some changes are out of scope or could be moved to a separate PR; some edge cases don't happen in practice and should just fail noisily instead of writing more code to maintain. When these are the only issues it's raising, then I know it's done.

I've tried and tried to get an LLM to delete code, it couldn't do it. I knew a file was 40% bad so what I ended up doing was deleting the file, then asking AI to create the missing file. That's how I got AI to delete code :)

It is surprisingly hard to do in a single prompt. I’ve had good luck though with being very explicit about asking for review, then to think deeply about what actually matters, and then reduce to a very short extremely concise reply. —- the model NEEDS to output those pages of text as part of its thinking process. It’s used to doing it in the assistant reply but can be cajoled into doing it in the thinking tokens.

Yeah, that's just a problem with review. I tend to keep it going until the stuff its finding is stuff that I look at and think "I can live with that". Usually by that point it's found the worthwhile stuff and what remains is minor or, like you said, yak shaving.

What tools are you using? I've found that Codex will happily tell me there's nothing more to do after a few rounds of /review and fix (or sometimes, it picks up on low value points that are technically correct but not worth changing).

My guess would be that this is a matter of fine-tuning the model to the problem. You would need to run the review experiment N times, observe the distribution of answers, and then fine tune with model operational parameters.

Perhaps someone more knowledgable could jump in here to clarify?

I’ve found stating “only call out critical issues, report but ignore everything else” will basically get what you want. Put that in a review skill and you’re good to go.

To be clear I mostly only use Opus and Gemini Flash but this might work for others too.

The verbosity itself seems to be a problem with post-4.6 Claude rather than a general issue with all LLMs. and IME the yak shaving is due to an overly generic prompt. We have a generic automated review bot but it’s prompted to only look for errors and never suggests refactoring; that was a conscious trade off to avoid what you’re talking about. If you’re reviewing interactively then you can give additional guidance about any code smells etc. that jump out