There is no taxonomy of these things you could make that will not cause some eyebrow raising choices.

It counts Clarence House in London in its top 100 as a world famous ‘palace’ - it’s a Georgian townhouse.

Age of sail trading post forts seem a relatively uncontroversial inclusion in comparison.

The title of the page is "Map of the world's great castles & fortresses". So those are latter.