Do we actually know enough about neural networks to be able to say how many parameters you need for a given performance? I’m under the impression that we’re still in the phase of trying things out and we can’t rule out different architectures massively reducing computational needs.

Parameter count is basically storage. Including persistent like flash memory. Which (still) is subject to Moore's law.

That's discounting architectural innovations & improvements which are bound to happen somehow.

In 5 years phones can easily have 1 TB flash