> In my experience these self-given-labels just express the views of some founding members and are often used to clarify who they do not want [...] and who is welcome [...]

This is where I think the problem is.

Once you start appending political identifiers then the purpose of an organization becomes more than just about X, but X according to certain values to the exclusion of others. There's nothing wrong with that but I could see how it can be viewed as disingenuous when it's insinuated that the organization is more open/general than it is apparent.