The problem is how long and what you have to do to get that 3-5 million number. No one who is drawn to the “passive income” hustle is thinking “work a normal job for 30 years, live under my means, and invest everything I can”. They want to get much more immediate results so they can enjoy life on easy street because grinding it out for so long sounds extremely depressing.
What you describe is a retirement plan, not a passive income lifestyle. Kinda the opposite of escaping.
This is literally the traditional definition of passive income - using your capital to generate more capital.
There is no free lunch, you need to provide something to get $$$. If you are providing labour it is by definition not really passive. That leaves land or capital.
So I moved to a cabin in the woods in a country with a low cost of living, and stuck pretty much all of it in the markets.
Had I not done that, I would have had to go back to work - instead I lived a modest life (€500/mo, max) off the income from putting my apartment on Airbnb, and regained my sanity after a decade of relentless work while my investments did their thing.
Anyway, it’s a decade on, still haven’t done a jot of “work”, and the assets are now worth several million, and are being redeployed to continue to maximise value growth - and we now treat ourselves to spending months travelling at exorbitant budgets, real estate, expensive toys - and had enough stability to decide to have a kid.
So yeah, it’s possible - although had we grown at 6%/yr rather than an average of 80%/yr, it would be a different picture - but I firmly believe there are plenty of other opportunities for rapid capital growth elsewhere in the markets, and yet to come. I’m just some average dude who buys equities on vibes and then sits on them for a decade. If I of all people managed it, others can.
Real estate. Historically, that's the way to escape work. It helps to inherit it.
25x expenses in s&p 500 works ok. (Adjust the multiplier for your level of pessimism) Funding it isn't easy, but save a good amount of your income for a few decades and control your expenses and you can get there.
EDIT: I'm also kind of writing in the context of having your own little economic engine that you own and control, and can be continually running, rather than owning a tiny piece of the abstracted aggregation of an entire economy's engines. That said, dead-simple, low-fee, market-indexed funds are a generally good place to put the surplus fruits of your own little economic engine.
That's not to say that passive income is impossible, its just not going to work if large swaths of the population are doing it.