This is like a type theory question - do you like it untyped or typed?

In physics, values have units too. Analogously, you could say - why incorporate units into the algebra in physics (as is often done)? Why not just add scalars etc. and not bother carrying around the units everywhere?

Well, because doing anything else is mostly nonsensical - it does not make sense to add meters and seconds together. Using unit algebra is the most basic sanity check as to whether your formula makes any sense.

Sometimes it makes sense to convert/cast between representations, but that should be explicit - distinguishing eg. objects and operations is more readable and more safe, and only comes with a bit of notational overhead. Nothing is free, but I think the benefits far outweigh the downsides.

One finds in regular vector algebra that "position vectors" and "displacement vectors" are sort of two distinct types of objects, and that it is never physically valid to add two position vectors together unless you create an affine combination like (a+b)/2. A position vector 'a' is really 'O + a', so [(O+a) + (O + b)]/2 = O + (a+b)/2, another position... but a+b on its own would really be (O +a) + (O + b) = 2O + a + b, which is not geometrically meaningful. So positions and displacements might both be elements of R^2, mathematically speaking, but there is something physically different about them, which physical applications/geometry forces you to contend with. I think it is something like a historical accident that there's not a great notation for expressing this in normal mathematics (or at least, I'm not aware of one!).