We had exams where the calculator was allowed, because we were tested on our ability to type in the right calculations. The calculator wouldn't save you.

If you can't work in order of magnitude calculations and estimations you're not going to last long in physics. Many useful models give good answers only within a certain range of inputs, while at the same time you often can tell where in a dynamic space you are by external observations. For example, estimating Martian wind speed within a factor of 3x from the size of the sand dunes, can be done having only seen pictures and with only the knowledge of a few dimensionless parameters.

That's also how you can tell if your calculator is "lying" to you (or you typed wrong). I guess I have a few similar tools in spaces where I'm more experienced to see when "hallucinations" and gibberish are being generated by LLMs. Of course having it "check sources" and evaluate its own solutions sometimes also works, if those are reliable, but you're on the hairy edge at that point.

I see an AI as an advanced calculator, but I see even a simple calculator as a poison to education if not strongly controlled.

Number sense is such a nebulous concept, but it captures what I've seen people struggle with. A general sixth sense about where an equation should or shouldn't go, a feeling that random numbers don't appear quite random enough and might have a pattern, or even recognizing that having a pattern signifies some relationship that otherwise has no evidence.