If the second LLM is just describing the content of the commit doesn't that defeat the purpose of the description, to capture the context that doesn't make it to the code?
replyThe second LLM is prompted to actually research the code, not just the diff, with fresh eyes to figure out what it does and why, before writing the commit message.
replyBut the code doesn't always have the answer to "why". At best that means the commit-writer has 'guessed' at why.
replyThis would have an even greater loss of the “why” context the author is describing in the piece.
replyThe human knows the "why", so they can correct the commit message if it is wrong or incomplete. Most of the time, though, they don't have to do that.
replyThis makes no sense to me. The code is already self-documenting if written well, and all you need is a one line commit message to summarize that.
replyDoesn't the original conversation at least retain the context about why the change was made? A different LLM literally has no way to tell why you made this change besides guessing from the codebase and git history.
If a change makes sense, a different frontier model can usually figure out why it was made from the code alone. I take that as a signal that that the commit is good.
replyI believe they can do this due to having millions of public pull requests and github issues in their training data.
There's pretty much always several plausible reasons for why a change was made. What's interesting is knowing exactly which one, especially when it later turns out to be wrong!
replyWhat a change does is self documenting. Why it was made and why that specific approach was chosen could be due to many things, such as:
reply* business objectives,
* the result of experimentation,
* the result of an offline conversation
etc.
This cannot be captured in code alone. This is why we write commit messages.
the mechanism is self documenting; context is not unless you pollute all your files with an ADR's worth of alternatives.
replyThe blog post is advocating against this.
replyIf you ask an LLM to write a message that explains "the why", it'll make up a why.
reply