Unless I'm misunderstanding your situation, I think you should look up the "Evil Maid Attack" to better understand how to mitigate risk for your threat model.
but linux is not as secure as an iphone, and linux users typically dont know how to set this up, so in practice you are right, it doesnt protect you
For me, a zero friction way to have decent security is worlds better than the normal state where homeservers are not encrypted at all.