I’m not sure I’m representative of “most people” in this respect (I have always used both n and m dashes), but I personally find the difference between n and m dashes bigger and more noticeable than the difference between regular and n dashes.