But the truth is, if you scale up as a company, it will end up costing you more in engineering salaries to re-create and maintain all the auth features you will need to support like @mooreds mentioned above than just using an auth library/vendor in the first place. And if you don't scale up, you are unlikely to push outside of the cheap or free tiers of any auth vendor you do go with, so it's a wash. Every reputable auth vendor has a pretty generous free tier.
It feels scary to go this direction because what happens if you blow up and then it becomes ultra expensive and now you're "locked in". All the time I see developers put hundreds of thousands, or millions of MAU into an auth vendor's pricing calc then gawk over the projected monthly bill and decide to roll their own. But the reality is that this doesn't happen. By the time you have that many MAU, you're typically paying out many multiples of that in engineer salaries alone, and paying an auth vendor a fraction of a single engineer's salary to handle your auth is substantially better value than hiring a team to build and maintain your own from scratch.
We have churned our fair share of large company customers over the years, as has every other auth vendor. They have all been because they needed feature we didn't have, not because the price was too high.
For an MVP or a prototype, I think it's okay to use an off-the-shelf tool. For something serious that will have long-term legs, I would do it myself. I hear all of your concerns and arguments and agree there are a lot of footguns. But again, having spent the better part of my adult life using and interfacing with these tools, I have an innate understanding of how to model auth correctly (separate it from the user, separate users from an 'org' or 'team' entity, etc).
You said it though, 'it depends' is really the right answer here.
I've heard plenty of stories of folks moving from homegrown auth to a off-the-shelf solution, but that's because I'm in the off-the-shelf auth space.
It'd be super interesting to hear stories of folks who went the other way, and outgrew their service provider's auth.
This is why a lot of SaaS you use these days will come with a "default project" or "default team" that might just be 1:1 with your own person. But injecting that abstraction layer makes it super easy down to the road to allow other individuals to join or participate in the management of those entities.