Perhaps someone else will (or did) write up a how-to for support mesh networking in your homebrew router.
I don't live in a densely populated city.
Even if you aren't doing wireless backhaul you just rely on regular client behaviour to transition between APs, can enable 802.11r to improve this.
Enterprise "mesh" typically uses wired backhaul for performance and can help clients roam quicker with a controller (auth, not deciding to roam). Controller can also adjusts radio power so APs aren't talking over each other if they're too close.
Mesh isn't any magic, just regular wifi.