If an author of a PR just generated code with an LLM, the GitHub PR becomes an incredibly inefficient interface between a repository owner and the LLM. A much better use of the owner time would be to interact with LLM directly instead of responding to LLM generated PR, waiting for updates, responding again, etc.
replyAs a project maintainer, I don't want to interact with someone's LLM. If a person submits a PR, using LLM or not, the person is responsible for any problems with it. How they respond to review is a good indicator if they actually understand the code. And if they used a bot to submit the PR, I'd simply consider it a spam.
replyYep, the indirection through the PR author is almost always inefficient and error-prone unless the author is really knowledgable about the code (many aren't).
replyThe bottlenecks today are:
reply* understanding the problem
* modelling a solution that is consistent with the existing modelling/architecture of the software and moves modelling and architecture in the right direction
* verifying that the the implementation of the solution is not introducing accidental complexity
These are the things LLMs can't do well yet. That's where contributions will be most appreciated. Producing code won't be it, maintainers have their own LLM subscriptions.
This is the part people keep missing. The bottleneck was never typing code. It's understanding the problem deeply enough to know what the right solution looks like. AI made the cheap part cheaper.
replyI don't know, I've had good experiences getting LLMs to understand and follow architecture and style guidelines. It may depend on how modular your codebase already is, because that by itself would focus/minimize any changes.
replyI still think there is value in external contributors solving problems using LLMs, assuming they do the research and know what they are doing. Getting a well written and tested solution from LLM is not as easy as writing a good prompt, it's a much longer/iterative process.
replythat's already happening tbh. the real issue isn't hypocrisy though, it's that maintainers reviewing their own LLM output have full context on what they asked for and can verify it against their mental model of the codebase. a random contributor's LLM output is basically unverifiable, you don't know what prompt produced it or whether the person even understood the code they're submitting.
replyMaybe a future direction will be the submission of detailed research, specifications and change plans for feature requests. Something that can be assessed by a human and turned into working code by both slides.
replyI wonder if that is an opportunity to build an Open-Source platform focused on this, replacing GitHub as the collaboration platform of a time where code was valuable.
replySome sort of LLM audit trail is needed (containing prompts used, model identifier and marking all code written by LLM). It could be even signed by LLM providers (but that wouldn't work with local models). Append only standard format that is required to be included in PR. It wouldn't be perfect (e.g. deleting the log completely), but it might help with code reviews.
replyThis would probably be more useful to help you see what (and how) was written by LLMs. Not really to catch bad actors trying to hide LLM use.
This would be a useful feature to bake into the commits generated by agents. Heck you don’t even need to wait — just change your prompt to tell it to include more context in its commit messages and to sign them as Claude rather than yourself…
replyThe GPL talks about "the preferred form for modification of the software", and I'm starting to think that anything which involves any kind of LLM agent should be including all the text that the user gave to it as well. Prompts, etc.
replyOf course, even then it's not reproducible and requires proprietary software!
I think guiding the LLM to write code is easy for them to write code by themselves.
replyThe "interesting situation" is that maintainers are unable to cheaply distinguish slop from good contributions so they will simply stop accepting outside contributions.
replyThis will cut off one of the genuine entry points to the industry where all you really needed was raw talent.