The force comes in at the other end, where we beat up people for not having X dollars. If you beat up and murder people for not having foo and we all just take this for granted and pretend there's nothing we can do about it, then we have no choice but to assign blame to the part we can actually comprehend - the choice of who gets foo.

Even the staunchest libertarians disagree with your understanding. Embarrassed that anyone holds or ever held this view, they have left you behind.