It doesn't make sense for the reader to spend more energy than the writer spent on creating it.
Great way to summarize cultural "economics"
Couldn't put the words on this pattern but sometimes all I care about is that someone cared about.
This is why I hate "interacting" with bots, scripts or AI/LLMs. It just wastes my time, again and again and again. Oddly enough not all humans understand that. About two months ago, a german developer involved with ffmpeg, spam-slopped their mailing list with AI (it was an AI proposal for some change to ffmpeg in the future). He still does not understand why that is a problem.
Something similar happened with PolyMC to PrismLauncher but that wasn't about AI, it was an absentee owner who came back and deleted everything he said was woke.
A reproducer might not actually be useful because there is basically no way short of fancy hardware tracing to figure out what the reproducer is doing.
Trying to figure out what to do based on possible future scenarios has a place, but not here when we are talking about a concrete present problem.
The rational choice would be to cut your losses and stop reading at that point. Once you realize zero effort went into the prompt, there's no longer any reason to read the output. The age old truism still applies: garbage in, garbage out.
If you think it's worthwhile, close the issue with a comment: "please rework this and show your work next time". Otherwise just close it without commenting and move on.
In my daily work I see these cases. It does help that the bugs that are filed contain a test case and some analysis by the agent.
It would not help to get ten identical bug reports all saying "I asked my agent to find a bug by prompting it with "find a bug and produce a test case". It found a nasty bug and a really nice reproducer. I'm not including its output here. Good luck!"
If it is the case that the original machine has an intermittent hardware fault, that analysis is exactly what I'd expect of an current AI. Voluminous and confidently wrong. Also interesting that the AI itself doesn't note this... having been biased by the conversation to consider kernel bugs, in the "cross-machine data" [1] it blames the kernel and doesn't appear to consider alternate explanations.
It doesn't get mentioned much around here but one major issue with AIs today is that they fall into tunnel vision very easily and often need some help getting out of it. There's a structural reason for tunnel vision built into their context limits, and there's also situations like this, where if you bias them in a certain direction they can get monomaniacally stuck on it. You can also see it where they do things like try to do something on your system like view a certain file, then if for some reason they fail due to permission issues, if you don't stop them they'll go absolutely insane trying different ways to access the file, rather than just stopping and saying "Hey, I need help". Even if you prompt them directly to do that, it only helps, it doesn't fix the issue. I suspect the RHLF training they get to be better coding agents reinforces this behavior because in the coding quality tests they're rewarded for one-shotting all the various benchmarks, where that "bull in a china shop" approach works better than giving up. But I'd actually like them to give up a bit more often. I've had the same problem with some particularly go-getting junior devs, too... I appreciate the ambition and I look forward to harnessing it in other situations but I'd rather you didn't spend five hours to create a terrible work around to something I could have gotten you in two minutes. For the junior dev, it's OK, they take the feedback and adjust... the AIs never adjust.
[1]: https://github.com/dfoxfranke/ripgrep-3494-analysis#6-cross-...
The issue was alongside the API boundary (timezone shenanigans) but the agent invented a lot of reasons with reasonable looking arguments because of the tunnel vision that there should be something wrong inside the project.
You got none of that here. It’s just realms of text.
Then maybe that person should not do it? At least until they find the time?
Claude is probably not right about the root cause here and is probably bsing, I agree. But it's collected enough raw data to point some expert humans at the right interaction. I'd take this bs bug report and start asking Claude some questions that would guide it to a more plausible actual answer, but that requires a little more experience in kernel development.
Maybe because it isn’t needed. That’s the reason we have experts and professionals, because it’s more economical to use them than for everyone to start from scratch. It’s easier to go to a mechanic shops to fix my engine block than to try to do it myself. I’ve heard that a lot of shops charge more if you said an amateur try to fix things first.
It’s a lot of stuff, but that’s not how you experiment for an hypothesis.
The ai doesn't have the same incentives as a human. It can also be applied to the repro where it exists.
Your mechanic analogy is a bit inapt. Sure they're experts and can replace your engine much better than you can. But tracking down a non obvious system level bug like this one is much less repeatable and much harder because the problem has already been filtered through a lot of reviews and testing. As a system becomes mature, the bugs become harder and harder individually to track down.
> As a system becomes mature, the bugs become harder and harder individually to track down.
What I'm talking about is that experts knows the system mechanism more than amateurs, so any hypothesis and experimental setup will be more focused and thus more economical than any amateurish one. And there's the matter of knowledge not present in some docs or other forms, such as past experiences.
Even with LLM tooling, we've seen the rises of harnesses and helper tools instead of relying on generation for everything. Who would you trust to build such harness, a domain experts or some random guy off the street?
That's the trillion-dollar catch, isn't it. LLMs love to write 30 paragraphs about some plausibly-correct-sounding explanation that is just as likely to be completely fucking wrong as it is accurate. The bug might be real, but that doesn't mean this analysis is accurate, and trying to figure out where the LLM went off the rails can be a nightmare. If you can actually understand the bug, it doesn't take 30 paragraphs to explain it. I would throw this bug report into my junk bin if I were on the receiving end of it, and I say that as someone who will spend days troubleshooting any issue a user will help me diagnose even if it only happens on their machine.