Consider a Linux desktop distro: if every little binary (out of order of magnitude 1000) acted like the center of the universe with gigabytes of build environment and "opinions" galore instead of portability, builds would take much more resources than they already do and parts wouldn't necessarily work together.
Putting toolchains in perforce is how it works for lots of C++ shops, the setup instructions are “sync and hit build”, whether there’s a toolchain upgrade required or not
It's not impossible, there just isn't that much demand for it.