I don't mean to be a jerk but ... if you are one of the people I manage, you literally are employed (at least in part) to make my job easier. That's not the only thing that matters -- which is why we (like many employers) do still allow some remote work. But making management more difficult is absolutely an impact that a rational workplace would take into account.