Since everybody runs their own unique permutation of apt or dnf packages, adding as little as possible will keep you as close as possible to what distro maintainers test. There's even OSes like Fedora Silverblue or Bluefin or SteamOS that ship with a fully baked _image_ - where installing system level packages is strongly discouraged - which helps ensure predictability and stable upgradeability.
Homebrew packages also tend to be more recent (this depends on your distro of course) and don't require elevated permissions to install.
[1]: Other unprivileged package managers like Mise or Nix do the same of course
Honestly, I would settle for a custom prefix if it tells me exactly what packages will break and what won’t without having to read each and every formula recipe. That’s one thing that bothered me for a while and I did not have the willpower to explore that direction without having community support.