" can be inches or arcseconds from cartography.
' can be feet (of measure) or arcminutes from cartography.
These actually all have slightly different symbols, and the symbols for the left/right quotation marks are actually farthest from this approximation, even if they are the most frequent usage.
It’s always been wrong in some respect to use a single available symbol to emulate three or more different typographical tasks, but quotation marks may actually be the one where the emulation is the most wrong.