Why does being so deep in the gravity well pose an issue? If you are assuming the Dyson swarm is intended to go back up the well then sure, but that isn't necessary.

Could you elaborate? Why would being deep in the gravity well be a non-starter? I thought Mercury's proximity to Sol was a huge advantage because of the ample solar power which would make planet-side manufacturing easier.