Science has always had an empirical component separate from its theoretical one. For a long time in human history, science was mostly empirical. The periodic table is a great example of mostly empirical observation organized into a pattern. I think the science of the 20th century was the "triumph of theory" so many of us have forgotten what a more empirically driven STEM world is like.

Agreed, I keep thinking to myself that there's a huge mathematical question right in front of us today that is in exactly this vein - all the various nuances of why LLM's work so well is a mathematical question. As far as I know, it's not really understood beyond "we do this basic thing (that makes sense) to predict that a noun is followed by a verb, and then we scale it up a bazillion fold and it can contribute to mathematics research".

As a comparison, classical computing has been scaled up a bazillion fold too, and can do things which are absolutely miraculous, but every layer of abstraction is discretely understandable.

While studying neural networks empirically like biology is one possible approach, there's no reason for that to be the blessed approach other than a combination of inertia and current lack of understanding. It's been only 15 years since AlexNet. Scientists struggled for centuries to model atoms before developing quantum mechanics and later QFT as an accurate quantitative framework.

Also, while biological systems simply exist in nature, artificial neural networks are ultimately mathematical objects with various properties that have yet to be uncovered.

I really like this way of putting it