It only matters if you have supply chain attacks in your threat model. Given they are up 400x since 2019, they should probably be in almost every threat model. Most distros operate on the honor system and that is not going to survive the post AI world.
Distros that are not fully hermetic and don't have reproducible packages (and there are many layers of reproducibility) will certainly have issues, but that's not a huge problem as long as you have a documented path to getting back to the current state. It doesn't need to be the fastest path, just a verifiable chain of trust.
It is critical to encourage many independent verifications that it be as fast as possible that someone can go from a clone of our tree of pure source code to the exact release hashes we publish.
Adding any binaries to that means someone that distrusts us must now go build those past releases as well, and if they rely on binaries, they must build those past past releases as well. This approach would make verification time go up dramatically every release.