Hardware projects live and die on scale. The engineering and tooling costs are a similar order of magnitude whether you make 1000 phones or 1,000,000. If you can guarantee that you have an accessible market for a million devices, then you're starting to get into the region of scale where this would be an OK idea.
Mind you, that's a million users who are cool with all the design tradeoffs you had to make to ingress protection, software performance with modern android, and form factor in order to get your desirable characteristic.
The Punkt MP02 is at roughly the price point and "niche-ness" as the product you describe here, and that sold for almost $400. They could afford to build in about the same amount of functionality as a Nokia brick of yore (but with 4G radios!) for that price.
EDIT: I forgot to check the "removable battery" checkbox; with it you get zero matching phones. Maybe you should've checked that before assuming GP just can't search.
Not to end on such a negative note, foregoing a maimum height and the removable battery, Sony's Xperia 5 and 10 fit the rest of the requirements and are very good phones. Hard to find for sale in the last few years, though.
But I main the $900 pixel.
They are so similar its weird, but Motorola was slow with snapchat and the keyboard some time.
As a wifi internet device it would work but I'm not sure that's what OP is going for.
Can we do 2010s phones with 2020s battery tech and modems please?