If it is the case that the original machine has an intermittent hardware fault, that analysis is exactly what I'd expect of an current AI. Voluminous and confidently wrong. Also interesting that the AI itself doesn't note this... having been biased by the conversation to consider kernel bugs, in the "cross-machine data" [1] it blames the kernel and doesn't appear to consider alternate explanations.
It doesn't get mentioned much around here but one major issue with AIs today is that they fall into tunnel vision very easily and often need some help getting out of it. There's a structural reason for tunnel vision built into their context limits, and there's also situations like this, where if you bias them in a certain direction they can get monomaniacally stuck on it. You can also see it where they do things like try to do something on your system like view a certain file, then if for some reason they fail due to permission issues, if you don't stop them they'll go absolutely insane trying different ways to access the file, rather than just stopping and saying "Hey, I need help". Even if you prompt them directly to do that, it only helps, it doesn't fix the issue. I suspect the RHLF training they get to be better coding agents reinforces this behavior because in the coding quality tests they're rewarded for one-shotting all the various benchmarks, where that "bull in a china shop" approach works better than giving up. But I'd actually like them to give up a bit more often. I've had the same problem with some particularly go-getting junior devs, too... I appreciate the ambition and I look forward to harnessing it in other situations but I'd rather you didn't spend five hours to create a terrible work around to something I could have gotten you in two minutes. For the junior dev, it's OK, they take the feedback and adjust... the AIs never adjust.
[1]: https://github.com/dfoxfranke/ripgrep-3494-analysis#6-cross-...
The issue was alongside the API boundary (timezone shenanigans) but the agent invented a lot of reasons with reasonable looking arguments because of the tunnel vision that there should be something wrong inside the project.
You got none of that here. It’s just realms of text.
Then maybe that person should not do it? At least until they find the time?
Claude is probably not right about the root cause here and is probably bsing, I agree. But it's collected enough raw data to point some expert humans at the right interaction. I'd take this bs bug report and start asking Claude some questions that would guide it to a more plausible actual answer, but that requires a little more experience in kernel development.
Maybe because it isn’t needed. That’s the reason we have experts and professionals, because it’s more economical to use them than for everyone to start from scratch. It’s easier to go to a mechanic shops to fix my engine block than to try to do it myself. I’ve heard that a lot of shops charge more if you said an amateur try to fix things first.
It’s a lot of stuff, but that’s not how you experiment for an hypothesis.
The ai doesn't have the same incentives as a human. It can also be applied to the repro where it exists.
Your mechanic analogy is a bit inapt. Sure they're experts and can replace your engine much better than you can. But tracking down a non obvious system level bug like this one is much less repeatable and much harder because the problem has already been filtered through a lot of reviews and testing. As a system becomes mature, the bugs become harder and harder individually to track down.
> As a system becomes mature, the bugs become harder and harder individually to track down.
What I'm talking about is that experts knows the system mechanism more than amateurs, so any hypothesis and experimental setup will be more focused and thus more economical than any amateurish one. And there's the matter of knowledge not present in some docs or other forms, such as past experiences.
Even with LLM tooling, we've seen the rises of harnesses and helper tools instead of relying on generation for everything. Who would you trust to build such harness, a domain experts or some random guy off the street?
That's the trillion-dollar catch, isn't it. LLMs love to write 30 paragraphs about some plausibly-correct-sounding explanation that is just as likely to be completely fucking wrong as it is accurate. The bug might be real, but that doesn't mean this analysis is accurate, and trying to figure out where the LLM went off the rails can be a nightmare. If you can actually understand the bug, it doesn't take 30 paragraphs to explain it. I would throw this bug report into my junk bin if I were on the receiving end of it, and I say that as someone who will spend days troubleshooting any issue a user will help me diagnose even if it only happens on their machine.