The limitations of mobile devices are mostly due to regulation (power limits and spectrum allocation and other requirements) and not inherent to the technology of handheld devices. And despite this, you can increase performance almost arbitrarily if you are able to increase the size and power and number of the antennas on the other side.

Yes, the Gen3 Starlink Direct-to-Cell constellation won't replace cell towers in urban or suburban areas. But I believe it could replace them in rural areas.