You just need to cover the bulk of uses, not every last thing you might need to do. I think the light SUV + utility trailer is a pretty good sweet spot, even better for many things than a pickup truck on its own. Like have fun lifting 20 sheets of plywood up onto a pickup truck and back down, assuming you've even got one with a long enough bed to support that whole pile.
And sure I'd imagine larger trailers get to be a hassle to move empty by hand, but isn't that what having more equipment is for? And isn't that well into the tier where you need a truck for a tow vehicle regardless?