the most frustrating part with having to compile a custom kernel is the maintenance burden (packaging/updating etc.), and not the time it takes to build…
I had a similar issue with networking modules for calico (k8s cni) on both rpis and jetson boards…
well, the tricky detail here (which we do not mention in the post, our bad) is that we got the raspbian config (cp /boot/config ... .config && make oldconfig) which includes most modules, and that's why it took more.
But yeap, good point about using the -j flag, it really accelerates the build!