So it is not as though you and the Andromeda Galaxy are made out of matter that got flung out of a point explosion long ago so that now you have traveled a very long distance away from one another, it is more like "both you and the Andromeda galaxy sat still for 13.8 billion years but space expanded between you in that time, so originally you were right on top of each other along with everything else".
We can rewind the model until the entire observable universe was as small as a Planck volume, but we have abundant evidence that the universe is indefinitely larger than that so even "that time when our 98gly diameter patch of space was almost indistinguishable from a mathematical point" means little when even that "point" was still just one pinprick out of the smooth manifold of a larger universe which could have been stupidly large or infinite even that early on.
A bit of an odd thing to say, since "everywhere" implies there are multiple places to be, and at the instant of the big bang, there was only place to be. So it was both everywhere and at a single point: it was at all of the single place there was to be.
But your point (sorry) about expansion being from everywhere isn't specific to the big bang; space was expanding well after the big bang and it doesn't seem like the expansion ever had a "center" (at least, not since not-center places existed). It's expanding from everywhere. (But evenly everywhere? I have no idea. Hey, maybe black holes are like buttons in cloth, and it expanded everywhere except for where the buttons were holding things still at a rate relative to distance from the button. A brilliant hypothesis that explains exactly zero unexplained phenomena, at least none that I know of.)
One theory that might help you visualize an alternative is that the big bang was basically two 3D universes (floating in higher-dimensional space) slapping against each other really hard. That creates an explosion everywhere even if everywhere is quite big or even infinite.