Unfortunately, swift-build is what Xcode has been using for years now, and has major performance issues: https://forums.swift.org/t/target-dependency-graph-computati...
Perhaps using it in both SPM and Xcode will lead to improvements, but I'm not especially optimistic.