>If performance is the main difference, whatever that means, then basically Go should be reserved for when Rust and other lower level langs cannot be used due to some other constraint?

Or when performance is the main but not the only difference, and there are many other benefits.

>Say I am building some app that I know will be CPU-bound, why choose Go over say... Swift?

Because unless you're building for macOS/iOS, Swift is really a no-go, with lackluster support for other platforms. Plus slow to build and convoluted.

> why choose Go over say... Swift?

Language religious wars are silly: you should choose a language based on your constraints and personal tastes. If there's no clear advantage of one language over another for a given task - then all the options are viable, pick one and get on with solving the problem.