It is possible that only needing one tank rather than two can make up for the dramatic loss of Isp we see from an air-breathing engine and the air-handling structure, but no one has yet managed to demonstrate that, and the general consensus runs against it. I recall reading that HOTOL (https://en.wikipedia.org/wiki/British_Aerospace_HOTOL) calculations were actually driven by an extremely light structure estimate rather than the airbreathing engine, to the point where if you plugged a rocket engine in they would actually get more payload to space as a SSTO, because those aggressively light structure estimates were doing all of the work.
Therefore nobody is ever going to invest the tens of billions required to develop a rocket based SSTO.
If somebody develops an engine that makes air breathing most of the way to orbit feasible, this has a chance of competing a Starship style architecture.
For the reasons you espoused, this is highly unlikely. However "highly unlikely" is more likely than "never".
Atmospheric density reduces exponentially with altitude, which implies that you would need to go exponentially faster to maintain mass flow into your engines and lift over your wings. The truth is that breathing air only gets you a third of the way to space, at best, so you have to have a rocket, and now you're battling that complexity. If your space plane doesn't breathe air, it probably is just better to punch your way out the way conventional rockets do.
Of course, the rocket equation is logarithmic, so reducing the amount of mass you loft gives you an exponential gain. This is true for all propulsion systems to an extent (different constants) but getting into space is the hardest propulsion problem we face. A space plane may or may not be better in this regard (it's been a while since I've looked into that kind of thing, so no opinion either way) but imo the inherent complexity is enough on its own to kill the idea.
The general idea is that you can get much better results in terms of deltav if you can find at least part of the reaction mass from elsewhere without carrying it onboard. Even inert nitrogen is useful as a reaction mass. Another way to get a good result is to use separate sources of reaction mass and energy. Then use that energy to accelerate the reaction mass as much as possible, so that you get a decent deltav by the time you exhaust the reaction mass. This is what ion and plasma thrusters do.
However, the requirement of the high thrust disappears once you finish the vertical climb. There's no danger of falling back to ground once you reach orbit. What you need at this stage instead, is to add velocity (deltav) to the craft to change its orbit/trajectory. This can be done even at very low thrust, because you have all the time you need. The limiting factor now is that you have only a finite amount of propellant onboard. You want to add as much deltav as possible before it runs out. A high thrust doesn't help because the engine will simply consume the propellant faster and exhaust it before you get the required deltav. This is where specific impulse comes into play. The maximum deltav you can get is proportional to the specific impulse of the engine (see rocket equation for details). As you can imagine, high specific impulse is critical for space missions requiring high deltav, like the New Horizons spacecraft that imaged Pluto or the Parker solar probe (interestingly, getting to the sun is harder than escaping the solar system). Rockets/jets with low thrust and high specific impulse are called sustainers.
The general trend seen is that specific impulse drops off as thrust increases. For example, the space shuttle solid booster has Tmax = 15 MN, Isp = 268s, and space shuttle orbiter cryogenic engine RS25 has Tmax = 2.28 MN, Isp = 452s. Meanwhile, the NEXT xenon ion thruster used in the DART mission has Tmax = 236 mN and Isp = 4200s. Note that the thrust has changed from Mega newtons to milli newtons. You would hardly recognize it if the ion engine thrusted against your body.
Wrt. aerospike engine - sounds nice, yet hardware wise it is heavier than the classic engine, and just look at that large number of pieces - just all those small mini-engines - it is made of and compare to Raptor 3. And for the optimal expansion - i'm waiting somebody will add a dynamically adjusting telescopic kind of end section to the classic bell nozzle.
A napkin to illustrate. Lets say you add a Raptor and 80 tons of fuel plus oxygen for it. That will give you 100 seconds of excess impulse of at least 160 tons (240 ton of thrust minus 80 tons) at the beginning to 240 tons at the end, so roughly 100 seconds of 200 tons. To get 200 tons thrust you'd need 20 fighter turbojet engines capable of at least Mach 3 - that is cost, complexity and weight dwarfing that one Raptor engine.
For scramjet, assuming we got a decent one, napkin is about the same. The best, my favorite, is air-augmented - scram-compress the air and channel it on the outside of the hot bell nozzles of the already working rocket engines - unfortunately the scaling mentioned above comes into play for meaningfully sized rockets though it has worked great for small ones.
PS: I have seen early-stage (but successfully tested) scramjets being developed for this purpose.
We have to ask: what exactly is a scramjet vehicle delivering? It's enabling the use of air instead of liquid oxygen. But how valuable is this? LOX is the second cheapest industrial liquid after water. The fuel part of a rocket propellant combination typically dominates the propellant cost. If a scramjet launcher uses more fuel (especially hydrogen) than a rocket vehicle would, it will end up increasing propellant cost per unit payload to orbit. It will also likely increase propellant volume per unit payload to orbit, especially if LH2 is used (LH2 being just 5% of the density of LOX).
All scramjet launchers need a rocket to reach stable orbit (since a scramjet cannot produce thrust at apogee to circularize above the atmosphere. So one can ask, what the tradeoff between the delta-V this rocket provides and that of the scramjets? From what I've heard, all such trade studies end up optimizing to 100% rocket and 0% scramjet.
A scramjet stage will be very light compared to an equivalent rocket stage, since it carries only the energy source (fuel) and not the full reaction mass. If this scramjet stage is able to impart a velocity close to the orbital velocity by the time it reaches the upper atmosphere, the subsequent rocket stage will have much less work to do to get it into orbit. And that translates to much less propellants (including oxidizer) and much less mass in the upper stage. It's not necessary to collect oxygen from the atmosphere to see an advantage.
Obviously, the raising of the perigee at apogee is going to need this rocket engine again. There are no launcher concepts that depend purely on scramjets.
Minimizing fueled mass of the vehicle is a stupid thing to do. It's optimizing the wrong metric.
Scramjets also suffer from bad thrust/mass and thrust/$ ratios compared to rocket engines.
Overall scramjet launch vehicles are an example of pyrrhic engineering: even if one could make such a vehicle "work", no one would want it.
Its not the cost, its the mass you're trying to reduce. So far, the engineering challenges have made it unfeasible, but its not a surprise that people look at the hundred tons of LOX on a rocket and imagine exchanging it for payload (or aircraft style re-usability).