The posh parts of America are suburbs or outer neighborhoods, not the inner city which is where most visitors end up staying.

For example, when you visit the San Francisco Bay Area, the actual posh areas are Marin, the Tri-Valley, and the hill adjacent areas of South Bay suburbs (eg. Woodside, Atherton, Saratoga, Los Altos, Los Gatos, Foster City, San Mateo).

Your average Japanese or other foreign visitor isn't visiting or staying in those suburbs nor would they be able to afford the hotel fees for hotels in those areas.

This is a major reason why Japan and Euro-trips have become fairly common amongst younger Americans now - renting a decent 4 start hotel room or Airbnb in a posh area of Japan that is out of reach for most Japanese comes out roughly the same as a middle-of-the-pack experience in the US because an American median salary is double the Japanese or European one.