And the people using asbestos in their homes, or buying homes with asbestos, are endangering themselves. How likely is that an improperly installed or maintained asbestos is going to affect a whole neighborhood?
Similar to how if you're using a insecure program written in C, whoever finds the bug should tell you immediately - don't use that program.
[0] You might say that asbestos from 1 house would affect the whole neighborhood but that would be similar to how a smoker smoking in the streets would affect the neighborhood. Second-hand smoking and second-hand asbestos are bad but they're negligible compared to living with a smoker who smokes indoors with the windows closed or smoking yourself.