We only need the ability to make make 1 qubit with indefinite reliable / fully error corrected state. Quantum interconnects would allow for these to be connected into arbitrarily large quantum computers.
It's more akin to "we only need to make reliable transistors to make classical computers." With the invention of the planar transistor, going from 1 transistor to 100 or 100k was not that big of a step, comparatively.