Edit: I now realise that the site specifically says "Map of the world's great castles & fortresses", so I guess the site considers them fortresses, not castles. I retract my snark.
It counts Clarence House in London in its top 100 as a world famous ‘palace’ - it’s a Georgian townhouse.
Age of sail trading post forts seem a relatively uncontroversial inclusion in comparison.