Compare: "Sacramento is the capital." You could say Sacramento is a capital if the reference group includes Albany, Austin, Tallahassee, etc. Or you could say Sacramento is the capital if the reference group includes Los Angeles, San Diego, San Francisco, etc.

Speaking of the house or town, the important information is what class it belongs to, not the specific castle/pool/roof...

As you point out, "the" will have a very specific, highlighting meaning in that case.

Yeah. I think what happens is that the implied group of things in which the castle belongs changes. "The town has the castle" could be understood as “there is only one castle and the town has it” (the group is that of all castles). But it’s nonsense. Your understanding seems right to me, but I am not a native English speaker. It is something along the lines of “among the things the town has, the only notable one is the castle” (it’s the only castle in the set of things the town has).

What is tricky is that the group in which the castle is unique is implicit.