Modifying the ABI of a function requires being able to track down all of the call-sites of the function, which is less trivial than you might assume. ABI concerns also tend to baked in relatively early in the optimization pipeline because you just simply can't get the ABI wrong, and I can think of several instances where the ABI decision causes missed optimizations.

There is also the other issue that a good algorithm for optimizing a problem like register allocation tends to be super-linear (e.g., quadratic), and if you shift the model from "allocate on a per-function basis" to "allocate all functions", the N in the O(N²) goes from "size of function" to "size of program," which is now suddenly a lot more compiler time spent for very modest gains. If register spilling across a function call is a noticeable component of runtime, then you're probably better off inlining that function in the first place!

Yeah, that's what MSVC and GCC did on x86, called "custom calling conventions" on MSVC and the regparm attribute on GCC.

All of these were dropped on x64, on x64 (and ARM) you get standard calling conventions for just about everything with proper unwind tables for functions.

It doesn't really "cheat" on the registers unless it inlines a function entirely. LLVM has support for custom calling conventions and pragmas to specify them, this is used by GHC on Haskell and other things, but it's practically unheard of in "normal" C/C++ code.

If it were practical and if there were significant performance benefits (on modern heavyweight CPUs) to tailoring the calling convention on a per-function basis, I'd expect to see it done in optimising JIT engines like Java HotSpot. As far as I know, they don't bother.

I'm no expert but I suspect jcranmer's comment has it right that you end up doing cross-function register-allocation while foregoing the other benefits of just inlining. I also suspect the payoff would be minimal on modern heavyweight hardware. I can see it making more of a difference on a very minimal embedded processor, or if optimising for the smallest binary possible.

This is already done in the form of deciding whether or not to inline a function.