Compare: "Sacramento is the capital." You could say Sacramento is a capital if the reference group includes Albany, Austin, Tallahassee, etc. Or you could say Sacramento is the capital if the reference group includes Los Angeles, San Diego, San Francisco, etc.

Speaking of the house or town, the important information is what class it belongs to, not the specific castle/pool/roof...

As you point out, "the" will have a very specific, highlighting meaning in that case.

Yeah. I think what happens is that the implied group of things in which the castle belongs changes. "The town has the castle" could be understood as “there is only one castle and the town has it” (the group is that of all castles). But it’s nonsense. Your understanding seems right to me, but I am not a native English speaker. It is something along the lines of “among the things the town has, the only notable one is the castle” (it’s the only castle in the set of things the town has).

What is tricky is that the group in which the castle is unique is implicit.

You use either “a” or “the” when you are specifying a class of things. “Our town has a castle” could be any castle. “Our town has the castle” would be a bit more rare and sounds weird as a general statement, but is valid in certain circumstances. “The castle” here means that the castle is something, or connected to something we have already been discussing and I’m referring to it, so the “our town” must be the new information. Like maybe we are discussing what we should do when you next visit, and I suggest seeing some particular famous tapestries. You say “wait, isn’t that castle [in which the tapestry is displayed] in Edinburgh?” And I reply “no, our town has the castle.”

You don’t use any article at all when what you’re talking about is unique. For example, “our town has Neuschwanstein Castle.”

Edit: or there is something which follows, e.g. “our town has the castle where they filmed Braveheart.”