I don't mean to be a jerk but ... if you are one of the people I manage, you literally are employed (at least in part) to make my job easier. That's not the only thing that matters -- which is why we (like many employers) do still allow some remote work. But making management more difficult is absolutely an impact that a rational workplace would take into account.
Before working remotely (pre-2019) when managing teams in person, I found myself necessarily having discussions to get synced with folks. At my most recent role (and previous remote first roles), team members were excellent at providing updates on Github issues (the sources of truth for work items). Of course, this required buy in at all levels and trickling company objectives down through the program(s) and linking work items to OKRs etc. It was very obvious when folks weren't hitting objectives and easy to gather detailed written evidence of this.
And regarding getting to know folks. Most recent offsite was at a villa in Croatia where I got to both meet my team members and ended up getting to know them like friends. Now that I think about it this has happened at previous companies as well during remote offsites.
I wonder if it's field-specific. Sounds like there are multiple anecdotes across a wide distribution of outcomes.
I think you are confounded by the fact your most overeager overachievers are going to return to office no matter what.
The arguments for and against fall along these lines. For RTO: in favor of the company over the self. It is more "productive" by some invented measure to work in the office, so it is the correct choice damned any other factor. A total trump card to those with this logic, like arguing the sky is blue.
And then what is the for WFH argument but the following: in favor of the self over the company. Perhaps if one pushed as hard as they could, they could get more done. They could sacrifice their sleep. They could grey their hair, increase cortisol, have an early heart attack and die. But in that time, they'd get a whole lot more done for the company certainly. WFH argues that affordances toward the employee ought to be made and even favored. Things like having choice in where one might live, not being saddled with a commute costly in time or money or both, being able to parallelize tasks such as taking the two minutes to start the laundry machine then returning to the desk, being able to see pets and loved ones for more than a few fleeting hours at the end of the day, better food, the list of benefits pretty much endless and also bespoke to the worker in question.
To be pro RTO, you have to be able to sacrifice the self like an ascetic, to deny all these tradeoffs and to grant the company control over you, your family and life outside of work (as where you live and how your family has to then live is a factor with RTO), all to benefit the company over yourself. The company that will never show you loyalty, that will use the same logic you are using to return to work to one day fire you.
Among my peers, on the younger side, no one really likes working in the office at all. They all are stuck with it and would desperately like to not work in the office. I expect over time, RTO will die as the generations that are culturally inclined to put the company over themselves retire from the workforce.
How does their quantative performance compare? Is there an opportunity in the differential?