(phabricator.services.mozilla.com)
It's concerning that Intel don't seem to have been responsive to anyone with respect to this issue and it doesn't appear to have an official errata yet, although Raptor Lake was the Intel CPU with voltage issues and basically random bit rot so I suppose it's hard to tell if this is a silicon level errata caused by bad design or by some kind of post-manufacturing damage. Raptor Lake in general causes enough non-reproducible noise that I believe Firefox gave up on automated crash reports from it ( https://bugzilla.mozilla.org/show_bug.cgi?id=1975808 ).
EDIT: I read that Oodle article (which is SO good!) again and realized that their customer-provided reproduction of the bug was directly linked to boost clock speeds (the customer said that overclocking by 5% made it happen entirely reliably), so this is definitely not a "the architecture has a 100% bug in it" but rather some deeper issue with clock propagation that appears at edge cases.
It also looks like there's a slight difference in the unwanted effect both companies have reported, despite the bug being seemingly triggered the same way (mov touching the high byte):
- Oodle reports that a low byte is occasionally stored in the intended location.
- Mozilla's fix suggests that a full 16-bit value is stored instead, corrupting an adjacent variable! This could have much more serious consequences.
Technically, this could still be the same exact bug. I found no mention of the order the output buffer was accessed in by the Huffman decoder debugged in the Oodle report, and, since it was a contiguous buffer, it's easy to mistake an occasional out-of-bounds copy there for a copy from a wrong location. But if both analyses are correct, the behavior of high byte accesses on Raptor Lake is way less predictable than those fixes suggest. Haven't managed to find an official erratum from Intel.
"Microcode and BIOS code requesting elevated core voltages which can cause Vmin shift especially during periods of idle and/or light activity" (emphasis mine)
https://community.intel.com/t5/Blogs/Tech-Innovation/Client/...
Recall also that "Vmin shift" means "the minimum voltage the processor needs to run correctly goes up" so if the issue isn't addressed, that level of undervolt may stop working
When you degrade the CPU naturally needs higher voltages to be stable, until the point where it just breaks completely and no amount of voltage it help it. But if your CPU doesn't degrade because it hasn't been overdoing it on voltages then there'll be no issues for Vmin to shift.
As an anecdotal experience from someone I know that runs these in prod for game servers, limiting the CPU to 80°C and 1.4V-1.45V, 400A has been keeping them alive for years doing 24/7 loads. Maybe a bit lower on the voltage if one wants to be sure longer term, as they are fine with just mass RMAing these. There's also large amount of differences in the silicon quality between samples that can make one run cool and completely fine even at the old stock settings, and an another sample that'll have to pull say 1.5x the power for the same load and clocks having it degrade.
Vmin will creep up, and the headroom for undervolting will degrade. It will affect the high clocks first (they demand the highest voltage), which is why dropping the max boost multiplier a step or two can also work around it (at the cost of basically downgrading it to a cheaper processor)
https://www.pugetsystems.com/blog/2024/08/02/puget-systems-p...
"Write both dist bytes as a single 2-byte store. This avoids the `movb %ch, [mem]` instruction pattern (store from high-byte register alias) that LLVM otherwise emits when dist arrives as a wide register. That pattern triggers the Intel Raptor Lake CPU errata, causing silent 2-byte stores that corrupt the adjacent `len` byte."
This is worse than https://en.wikipedia.org/wiki/Pentium_FDIV_bug
https://news.ycombinator.com/item?id=27244941
Edit: you should probably read the article I linked first.
$ echo 1 > /sys/devices/system/cpu/microcode/reload
Just that it's writable by $ (not #) feels awkward.
sysutils/intel-microcode-netbsd
cpu 0: ucode 0xf0->0xf6
cpu 1: ucode 0xf0->0xf6This is a good question. As others have noted below, yes, and sometimes you can see kernel logging on start-up when the microcode is loaded.
The subject of CPU microcode update mechanisms is an interesting and relevant topic, but such a shallow, low-effort question is not a good way to promote interesting discussion on that topic.
See also “Eternal September”.
Regarding the Raptor Lake bug I received a couple of messages from confused users that had read articles on Tomshardware and Neowin. They asked about erratas and microcode updates which puzzled me, because that was part of my early investigation into the bug and we know that the failure is not caused by a known errata and microcode updates cannot fix broken CPUs. So why did they ask? As it turns out it was slop. Both articles are 100% slop full of confusing and inaccurate claims.
Because there still are many broken CPUs out in the wild. Firefox works around the crash so the broken CPUs won't flood the channel with crash reports.
For example:
Does he mean all existing 13th/14th gen CPUs (prior to Intel's discovery of the vmin issue) are broken in the sense that they are susceptible to damage and can only be replaced.
OR
Does he mean that the microcode updates, applied by Intel to existing CPUs that are susceptible to damage, will only slow degradation and the CPUs will eventually fail and can only be replaced.
OR
Is he saying the 13th/14th gen CPUs which have already sustained damage, cannot be fixed by microcode updates.
If the CPU is damaged already, the new microcode wont fixed the problem. It broken. You have to RMA the CPU.
The vmin shift instability is fixed. There are no new report of mass failures of 13th/14th gen CPUs after the new bios/microcode release.
Agner Fog's optimization guide says "Any use of the high 8-bit registers AH, BH, CH, DH should be avoided because it can cause false dependences and less efficient code."
It's actually pretty easy to get compilers to use those, you mainly need a bunch of narrow accesses to neighboring memory. The oodle post contains a godbolt link to pretty ordinary c code triggering this.
I'd guess that you also need some other conditions (multiple in flight stores, high boost speeds) to trigger this.
That's unfortunate, because it's precisely why things like this will keep happening.
Agner Fog's optimization guide says "Any use of the high 8-bit registers AH, BH, CH, DH should be avoided because it can cause false dependences and less efficient code."
The sad vicious cycle of compilers not exercising the hardware, and then the hardware designers not paying attention. Using the high 8-bit registers and "implicitly merging" them is one of the ways to reduce the number of instructions and thus improve size optimisation.
I have the opposite opinion. Its use being rare means CPU designers have less need to optimize for that rare case, and hardware optimizations are precisely where these kinds of issues tend to pop up.
And high 8-bit registers are a x86-specific feature, other CPU families don't have it. So that special case being less optimized (or even pessimized) is not much of a loss.
Intel blew it when they let them continue to work in to 32-bit code on the 386, and then AMD blew it when they repeated the mistake when defining the 64-bit ISA.
There could theoretically be instruction selection passes that are biased toward rare instructions, specialized for fuzzing hardware, I'm surprised Intel doesn't already do that.
Clearly Intel needs to do far more extensive regression-testing, with things like demoscene productions --- especially the extremely size-optimised ones that can exercise the edge-cases much better than the usual "compiler slop".
You’re intentionally muddying the waters with meaningless philosophy. Even the law makes the difference between “knowingly” (with knowledge, intention, premeditation) and “mistake”. They didn’t knowingly break the CPU but they knowingly launched it despite their own internal findings, and knowingly blamed others when this came out.
But here you are claiming that the company must deserve the benefit of the doubt on their intent.
No, “knowingly” is most definitely not “meaningless”. And anyone who’s not naive or bad intentioned should make that difference and take note. Every time a company gets away with it because an army of philosophers washes away any guilt or plays it down with meaningless distinctions it becomes one more reason to do it again. Knowingly.
I don't think they've clarified which exact models of which serial number ranges are affected with oxidation to this day. It should've been a recall.
This is why CPU designers should aim for simplicity. This is why RISC-V vector extension, which requires complicated logic, can become a source of implementation errors.
[1] https://edc.intel.com/content/www/us/en/design/products/plat...
Fortunately, most of the erroneous behaviors are triggered only by very unlikely combinations of circumstances, some of which may even be impossible to happen in user programs, but only in operating system kernels.
Nevertheless, from time to time there are also serious errata, like the one discussed here, which can be triggered even by ordinary user programs. Sometimes, like here, such errata can be avoided by compilers patched to not generate the buggy instructions for the affected CPU models, assuming that it is known with certainty on which model of computer the compiled program will be executed (or using code dispatch at run time, based on the CPU model).
Simplicity in the CPU hardware may reduce the probability of hardware bugs, but it increases the probability of software bugs, because the missing hardware features must be implemented at a much greater cost in software, like in the case with the missing integer overflow detection of RISC-V, which causes most RISC-V programs to omit overflow checks, increasing the chances of undetected bugs.
Since I've got a SpacemiT K3 board my self now, I though I test it again:
I compiled microjs with both tinycc and chibicc, which where both compiled for the target platform with and without -ftrapv:
Slowdown Zen1: tinycc: 1.34%, chibicc: -0.3% (slight speedup somehow?)
Slowdown X100: tinycc: 0.1%, chibicc: 3.4%