If you're looking for a more formal approach, ie the infinitesimal analogue to the usual real analysis, it's called nonstandard analysis and you could probably start with the original, eponymous book written by the creator, Abraham Robinson, for which I unfortunately don't have a link.
If this stuff interests you btw I would also check out Knuth's book on surreal numbers[3], which I believe, in some sense, are the fullest possible extension of what we think of as numbers? But it's been a while since I read into those.
[0] https://www.bravernewmath.com/ [1] https://intellectualmathematics.com/calculus/ [2] https://people.math.wisc.edu/~hkeisler/keislercalc-06-03-26.... [3] https://people.math.harvard.edu/~knill/teaching/mathe320_201...
Generally most of these 'handwavy' notations are rigidly provable, but only under general assumptions, that might not be true in special cases.