Generally this has not been my experience. Typically both very slow CPUs like the ATMega328 and high-performance CPUs like an Intel i7 are in the neighborhood of 1 nanojoule per instruction. (For operations more than 8 bits wide, the AVR is significantly worse.) Low-power CPUs like the STM32L line are commonly around 0.2nJ/insn. Ambiq's subthreshold logic gets them closer to 0.05nJ/insn.
I agree with the rest of your comment.
The fairest comparison is between chips from major vendors in the same year. If you're just doing FP32 matrix multiplies, a 2026 nVidia GPU is ~1 pJ / FLOP, an Apple M4 is ~30 pJ, and a 2026 Intel Xeon is ~50 pJ. The quality of process and optimization is a little different between those 3 companies, but not by as much as 2x. But from those numbers, you can reliably conclude that the GPU architecture is much better for matrix multiplies.
By this I mean: currently, transforming a program written in a typical high-level language to optimized ISA code involves what can only be described as black magic inside the compiler. If you don’t know the implementation details of modern superscalar OOO branch-predicting microarchitectures then you may have no clue why two very similar-looking functions could have dramatically different running times. Cue Matt Godbolt tech talk…
What I see as the more promising side of this type of development is a tighter coherence, and thus more transparency in expected runtime performance, between the software above the ISA and the hardware below it. Essentially we now have a linear machine with a linear source language, and with the popularity of Rust nowadays we actually have a critical mass of developers who are already used to applying ownership discipline in program design.