Personally I'm more worried about someone stealing the entire server or a local threat actor.
Sure, keep TPM to help with boot integrity, maybe even a factor for unlock, but things like Clevis+Tang (or Bitlock Network Unlock for our windows brethren) is essential in my opinion.
Unless I'm misunderstanding your situation, I think you should look up the "Evil Maid Attack" to better understand how to mitigate risk for your threat model.
but linux is not as secure as an iphone, and linux users typically dont know how to set this up, so in practice you are right, it doesnt protect you
For me, a zero friction way to have decent security is worlds better than the normal state where homeservers are not encrypted at all.