I've often been suspicious of this too, having noticed that building one of my projects on Apple Silicon is way quicker than I'd expect relative to x64, given relative test suite run times and relative PassMark numbers.

I don't know how to set up a proper cross compile setup on Apple Silicon, so I tried compiling the same code on 2 macOS systems and 1 Linux system, running the corresponding test suite, and getting some numbers. It's not exactly conclusive, and if I was doing this properly properly then I'd try a bit harder to make everything match up, but it does indeed look like using clang to build x64 code is more expensive - for whatever reason - than using it to build ARM code.

Systems, including clang version and single-core PassMark:

    M4 Max Mac Studio, clang-1700.6.3.2 (PassMark: 5000)
    x64 i7-5557U Macbook Pro, clang-1500.1.0.2.5 (PassMark: 2290)
    x64 AMD 2990WX Linux desktop, clang-20 (PassMark: 2431)

Single thread build times (in seconds). Code is a bunch of C++, plus some FOSS dependencies that are C, everything built with optimisation enabled:

    Mac Studio: 365
    x64 Macbook Pro: 1705
    x64 Linux: 1422

(Linux time excludes build times for some of the FOSS dependencies, which on Linux come prebuilt via the package manager.)

Single thread test suite times (in seconds), an approximate indication of relative single thread performance:

    Mac Studio: 120
    x64 Macbook Pro: 350
    x64 Linux: 309

Build time/test time makes it look like ARM clang is an outlier:

    Mac Studio: 3.04
    x64 Macbook Pro: 4.87
    x64 Linux: 4.60

(The Linux value is flattered here, as it excludes dependency build times, as above. The C dependencies don't add much when building in parallel, but, looking at the above numbers, I wonder if they'd add up to enough when built in series to make the x64 figures the same.)