Labor and materials isn't the problem with housing costs. It's onerous permitting and zoning and code requirements.
With that said, the advantage disappears compared to national builders- the guys who buy up big farm fields and build entire subdivisions all in one go before they even have buyers. They can keep crews rotated between jobs in a fairly predictable schedule, so the only thing holding anyone back compared to the factory is bad weather preventing digging out room for foundations.
you as a one off can't because the plumber isn't going to give you the priority needed.