Quoting from the bug analysis:
>Headline. The crash is real and reproducible. With musl instrumentation we pin the in-process mechanism precisely: a thread's own store to a freshly- faulted anonymous page becomes invisible to that same thread's reload ~10 instructions later, because the page's backing is replaced mid-function. A pagemap read at the instant of the fault shows the backing is the kernel's zero page. A captured core dump confirms the crash site with matched virtual addresses. The mechanism localizes to the interaction between the per-VMA-lock anonymous-fault fast path and a concurrent munmap's TLB shootdown. A source-level review of Linux 7.0.12 identifies a specific race in that interaction, and a git comparison across v6.19/v7.0/v7.1/mainline identifies the v7.0-introduced change on the munmap-teardown side that widens it.
"a thread's own store to a freshly- faulted anonymous page becomes invisible to that same thread's reload ~10 instructions later, because the page's backing is replaced mid-function." -> ??? What's a "backing" of a page? Freshly-faulted? Fresh fries? "~10 instructions later"?
"A pagemap read at the instant of the fault shows the backing is the kernel's zero page." -> Backing?
"The mechanism localizes to the interaction between the per-VMA-lock anonymous-fault fast path and a concurrent munmap's TLB shootdown." -> How can a mechanism "localize"?
And more.
This is... words strung together. Nothing more. I wonder how people read and make sense of this.
In case people have forgotten what real technical writing looks like, here's a sample (I am not the author): https://yifan.lu/2019/01/11/the-first-f00d-exploit/
The worst part is how compulsive LLMs are at writing like this. Like the system prompt instructs it to "reason" and like a college Sophomore it pontificates and quotes Nietzsche to fein intelligence and orginal thought.
I understand that it's probably a hard technical problem to get models to conform to a style without sacrificing performance in other areas. But I really wish AI companies would spend more effort getting them to write in a neutral, concise way and not like a middle manager on 80mg of Adderall.
I wonder if this would make it harder or easier to detect whether the underlying meaning of the writing is bullshit.
MMU: The memory management unit, part of the CPU designed to allow an OS kernel to flexibly control how memory addresses used by a program map to physical RAM.
Page: A 4k region of memory addresses.
Anonymous: The page is just regular old memory used by a single process for general purposes, it's not part of e.g. a memory-mapped file.
Backing: The memory corresponding to a page. The kernel can direct the MMU to map any page to any part of RAM, or delete the mapping entirely.
Zero page: The kernel keeps a single 4 KB page filled with zeros (the "zero page").
Page fault: An error that occurs when you try to write a page that's not present, or write to a read-only page. Some page faults are normal and expected by the kernel, they support features like swapping, memory-mapped files, etc. Such "expected" page faults are invisible to your program. (On the other hand, an unexpected page fault occurs when your program just straight-up accesses an address it's not supposed to, which will end your program with a SIGSEGV signal -- an all-too-familiar experience for a C programmer.)
When you allocate say 4 MB of memory, the kernel simply sets it to 1k read-only copies of the 4 KB zero page. Then when your program tries to write to a page, it faults. The kernel handles this expected fault by assigning that page an individualized memory region. If a program allocates a lot of memory, it's only assigned memory for what it actually uses, when it first uses it. This is an optimization: Every program that allocates more memory than it needs is wasteful, but the kernel can recover that waste by not backing the allocation with memory -- the program has to prove each page is needed by writing to it.
Store to a freshly-faulted anonymous page: The program stored data to its own memory, which caused an expected page fault.
> becomes invisible to that same thread's reload ~10 instructions later
The thread fairly quickly tried to read the data from the same place in memory it had just written to. It doesn't get the same data back. This is a major problem that causes the program to malfunction and crash.
> A pagemap read at the instant of the fault shows the backing is the kernel's zero page
The expected fault occurred because the program tried to write to an address that was mapped to the zero page, probably because it's the first write to freshly allocated memory. (This rules out other reasons it might be faulted, e.g. the page had been swapped to disk.)
> concurrent munmap's TLB shootdown
The page table is big and slow so the CPU caches parts of it in an area called the TLB (translation lookaside buffer). munmap is a kernel function that removes the backing for an address region. Since munmap changes the mapping, it has to inform the MMU the relevant part of the TLB is no longer valid.
> per-VMA-lock anonymous-fault fast path
This is what the kernel does when the program first writes to a zero page. Saying it's the "fast path" implies there's some (slow) special case handling in the kernel code, but the bug doesn't trigger any of the special cases, we're in the most common case, which the kernel code tries to handle as quickly as possible because it has measurable impact on performance. I would guess per-VMA-lock has to do with how the kernel synchronizes this code (a "lock" is a basic synchronization primitive, you should be familiar with it if you do any kind of multithreaded programming).
> The mechanism localizes to the interaction between the per-VMA-lock anonymous-fault fast path and a concurrent munmap's TLB shootdown. A source-level review of Linux 7.0.12 identifies a specific race in that interaction
The mechanism: What's actually happening to cause the program not to be able to load data it just stored in memory.
Localizes: The cause of a problem like this is a needle in a haystack -- it could be caused by the program, the kernel or the hardware. The analysis has narrowed down the haystack to a specific part of the kernel code.
Race in that interaction: Two parts of the kernel code, (1) The kernel code for munmap and (2) the kernel code to handle a program's first write to the read-only zero page for a fresh allocation. These two parts work fine individually but the problem occurs when they both try to change the page tables / TLB at exactly the same time. This is quite a small, specific haystack compared to "somewhere in the program, kernel or hardware" we started with.
I’ve been reading the LWN kernel page for >20 years so I’ve picked up almost all the jargon and could understand it pretty decently.
You’re dead on in saying it was written for that specific audience, not anyone more “normal”.
> a thread's own store to a freshly- faulted anonymous page becomes invisible to that same thread's reload ~10 instructions later, because the page's backing is replaced mid-function
A result of a machine instruction to store something in RAM got erased, when the same thread attempted to load it ~10 machine instructions later. The reason is that the physical RAM page backing that particular virutal page got replaced while the thread was running.
The fault happens in the fast path for anonymous (i.e. not backed by files on disk) pages, in the granular per-VMA (virtual memory area) locks.
> "A pagemap read at the instant of the fault shows the backing is the kernel's zero page." -> Backing?
Yes, it's typical kernel terminology. "A backing page" or a "backing file".
> This is... words strung together. Nothing more. I wonder how people read and make sense of this.
There's nothing wrong with this description. It's just written for kernel developers. It needs to be expanded and explained for people who are not.
The "backing" refers to the physical memory that might or might not be present for every "virtual" piece of memory that your program has allocated.
"Freshly faulted" means that a page of (virtual) memory has just received a "backing" by the process above and is, thus, very fresh in physical memory (even though the virtual memory might have been allocated much earlier)
"~10 instructions later" refers to the assembly- (machine-) code, which, contrary to a high-level language like C, usually has long(ish) sequences of rather simple "instructions". 10 instructions is a rather short interval in assembly code.
As for the "localize", the term used is actually "localizes to" which I read as "turns out to be located in" (probably just a bad English translation by the original author)
While this whole summary reads a bit weird, I don't think it is necessarily the result of an LLM, it's probably just that someone who is rather inexperienced at writing up technical summaries did it.
I’m guessing it wrote to a page and read back zero page as the memory was in the mist of being allocated.
The write-up should have been about five paragraphs: (1) "The script at https://... generates a tree with X million files taking 20 GB total. Running ripgrep 1.2.34, compiled with musl, on my Threadripper 9876 as 'rg ...' crashes about a fourth of the time. With glibc, it does not crash." (2) "This appears to be a bug introduced by commit abcdef1234 in some_vm_call() that allows a race between [...] such that user space can see a corrupt whatever." (3) "The sequence of operations that causes this is: (show two or more kernel threads with lines interleaved to explain the bug)". (4) and (5) as needed to elaborate on those three core points.
But the actual post is long on (fluent) speculation and short on reference to actual code. That's a large part of why it's offensive slop.
So "backing" is the underlying physical memory the page maps to.
So if I were to make sense of that piece of slop, is this what it would be?
"In one thread, a page fault happens during a store operation. The physical memory address is the proper address, which is immediately (around 10 assembly instructions later) replaced by the 0x00000000 memory address by another thread due to race condition, resulting in a crash when the original thread tries to read that page again."
Reading from the zero page is legal, that doesn’t cause it to crash. The crash is because of a logic “bug” in the program where the zero it reads back causes some other issue in the program.
I say “bug” because it’s clearly impossible without the kernel messing up. It just stored a non-zero value there.
What happens here is a bug in munmap() that happens in another thread. It corrupts the TLB, for a brief instant replacing the correct page.
So that the virtual memory page that the crashed thread tried to read its RAM, it gets replaced by the special zero-filled physical page. The kernel uses this zero page as an optimization when it needs to create a region of data filled with zeroes, so it can just map one physical page over the entire range.
It's a really low-level bug that just requires a lot of specialized knowledge just to explain. A better write-up is certainly possible but needs to have a lot of explanations to make it understandable.
Translation: if this nails-on-chalkboard LLM spew identifies a kernel bug, which is then patched, its aesthetic qualities do not interest me in the slightest.
The only people who have a need to judge the quality of the report (not aesthetically, I mean) are those few who know enough about the subsystems of the kernel it implicates to do something about it.
My point: does it lead to a bugfix? Good. No? Bad.
Do you have an answer to that question? I believe it's a bit early, no?
The overflow would still overflow; the use-after-free would still use after free; a musl mask race would still race.
Just happens to be that the musl code is able to hit this and other code isn’t for some reason.
I don't follow. An application might see this kind of crash if it has a bug causing it to access a page while another thread is mapping or unmapping that page. That would be a bug in mallocng, musl or ripgrep. Or, as someone else mentioned, it could be bug in the processor's virtual memory logic that has the same effect. Why do you say it can only be a kernel bug?
> Headline. The crash is real and reproducible.
anyway it's slop, can sense it even before i started reading
(Feel free to pretend that I used some phrase other than "intellectual work" if you dislike seeing it used to describe something done by AI.)
So having the prompt available would be useful even in the degenerate case.
- Mimic human language (and logic, since that's part of what we express with language). This includes code written by humans.
- Write code to solve problems. The figure of merit here is solving the problem, not reproducing anything human.
I'm not an expert on LLMs, but it's not obvious that the human reasoning they are fitting in the first case is going to be anything like the problem solving required in the second case. It was trained to get itself out of a problem, not to do it in a way that a human would relate to.
And we're already running out of human data to train on, while synthetic data has no limit. Bug reports in the future might amount to "fix this because then I'll get a cookie".
This sounds to me like either (a) a complex race involving a CPU migration at an awkward time or (b) a bug in the zap path transiently exposing wrong PTEs.
Also, I don’t think the zero page has pfn zero.
If I had to throw a dart, I would guess that direct page table zapping is allowing a CPU to read through a higher-level-paging-structure cached entry to a table that has been freed and reused. Yuck. I’ve debugged one of these before.
I think the broad strokes are that, due to bad locking in the kernel, mmap() returns a VA that a concurrent munmap() is still in the middle of unmapping. The specifics beyond that seem murky/speculative/inconsistent.
Alternatively, it could be a hardware bug, since afaict it's only been repro'd on one hardware config. (I know there are a bunch of ARM cores with erratas around TLB invalidation)
https://lore.kernel.org/all/CALCETrXbj__SFQMzPZhES5y6-sh4np-...
Which raises the question of whether any HN readers have successfully reproduced this?
I've just tried (using his file generation script and the official rg binary he links) on a Ryzen 7 5800X / 7.1.5-arch1-2 with sufficient free ram as the github issue suggests, and still no segfaults after 10 minutes.
With this particular issue, if my theory is right, it will depend on all manner of microarchitectural issues, possibly address spare randomization, context switch and migration timing, random speculative accesses that pull things into cache, etc. Yuck.
It doesn't make sense for the reader to spend more energy than the writer spent on creating it.
Great way to summarize cultural "economics"
Couldn't put the words on this pattern but sometimes all I care about is that someone cared about.
This is why I hate "interacting" with bots, scripts or AI/LLMs. It just wastes my time, again and again and again. Oddly enough not all humans understand that. About two months ago, a german developer involved with ffmpeg, spam-slopped their mailing list with AI (it was an AI proposal for some change to ffmpeg in the future). He still does not understand why that is a problem.
Something similar happened with PolyMC to PrismLauncher but that wasn't about AI, it was an absentee owner who came back and deleted everything he said was woke.
A reproducer might not actually be useful because there is basically no way short of fancy hardware tracing to figure out what the reproducer is doing.
Trying to figure out what to do based on possible future scenarios has a place, but not here when we are talking about a concrete present problem.
The rational choice would be to cut your losses and stop reading at that point. Once you realize zero effort went into the prompt, there's no longer any reason to read the output. The age old truism still applies: garbage in, garbage out.
If you think it's worthwhile, close the issue with a comment: "please rework this and show your work next time". Otherwise just close it without commenting and move on.
In my daily work I see these cases. It does help that the bugs that are filed contain a test case and some analysis by the agent.
It would not help to get ten identical bug reports all saying "I asked my agent to find a bug by prompting it with "find a bug and produce a test case". It found a nasty bug and a really nice reproducer. I'm not including its output here. Good luck!"
If it is the case that the original machine has an intermittent hardware fault, that analysis is exactly what I'd expect of an current AI. Voluminous and confidently wrong. Also interesting that the AI itself doesn't note this... having been biased by the conversation to consider kernel bugs, in the "cross-machine data" [1] it blames the kernel and doesn't appear to consider alternate explanations.
It doesn't get mentioned much around here but one major issue with AIs today is that they fall into tunnel vision very easily and often need some help getting out of it. There's a structural reason for tunnel vision built into their context limits, and there's also situations like this, where if you bias them in a certain direction they can get monomaniacally stuck on it. You can also see it where they do things like try to do something on your system like view a certain file, then if for some reason they fail due to permission issues, if you don't stop them they'll go absolutely insane trying different ways to access the file, rather than just stopping and saying "Hey, I need help". Even if you prompt them directly to do that, it only helps, it doesn't fix the issue. I suspect the RHLF training they get to be better coding agents reinforces this behavior because in the coding quality tests they're rewarded for one-shotting all the various benchmarks, where that "bull in a china shop" approach works better than giving up. But I'd actually like them to give up a bit more often. I've had the same problem with some particularly go-getting junior devs, too... I appreciate the ambition and I look forward to harnessing it in other situations but I'd rather you didn't spend five hours to create a terrible work around to something I could have gotten you in two minutes. For the junior dev, it's OK, they take the feedback and adjust... the AIs never adjust.
[1]: https://github.com/dfoxfranke/ripgrep-3494-analysis#6-cross-...
The issue was alongside the API boundary (timezone shenanigans) but the agent invented a lot of reasons with reasonable looking arguments because of the tunnel vision that there should be something wrong inside the project.
You got none of that here. It’s just realms of text.
Then maybe that person should not do it? At least until they find the time?
Claude is probably not right about the root cause here and is probably bsing, I agree. But it's collected enough raw data to point some expert humans at the right interaction. I'd take this bs bug report and start asking Claude some questions that would guide it to a more plausible actual answer, but that requires a little more experience in kernel development.
Maybe because it isn’t needed. That’s the reason we have experts and professionals, because it’s more economical to use them than for everyone to start from scratch. It’s easier to go to a mechanic shops to fix my engine block than to try to do it myself. I’ve heard that a lot of shops charge more if you said an amateur try to fix things first.
It’s a lot of stuff, but that’s not how you experiment for an hypothesis.
The ai doesn't have the same incentives as a human. It can also be applied to the repro where it exists.
Your mechanic analogy is a bit inapt. Sure they're experts and can replace your engine much better than you can. But tracking down a non obvious system level bug like this one is much less repeatable and much harder because the problem has already been filtered through a lot of reviews and testing. As a system becomes mature, the bugs become harder and harder individually to track down.
> As a system becomes mature, the bugs become harder and harder individually to track down.
What I'm talking about is that experts knows the system mechanism more than amateurs, so any hypothesis and experimental setup will be more focused and thus more economical than any amateurish one. And there's the matter of knowledge not present in some docs or other forms, such as past experiences.
Even with LLM tooling, we've seen the rises of harnesses and helper tools instead of relying on generation for everything. Who would you trust to build such harness, a domain experts or some random guy off the street?
That's the trillion-dollar catch, isn't it. LLMs love to write 30 paragraphs about some plausibly-correct-sounding explanation that is just as likely to be completely fucking wrong as it is accurate. The bug might be real, but that doesn't mean this analysis is accurate, and trying to figure out where the LLM went off the rails can be a nightmare. If you can actually understand the bug, it doesn't take 30 paragraphs to explain it. I would throw this bug report into my junk bin if I were on the receiving end of it, and I say that as someone who will spend days troubleshooting any issue a user will help me diagnose even if it only happens on their machine.