But the people in each house would suffer from from asbestos in their own house. [0] If I bought a house in such a neighborhood, I'd have it checked from cracks or deterioration. If the results show a risk, I'd inform the neighboring houses, too.

Similar to how if you're using a insecure program written in C, whoever finds the bug should tell you immediately - don't use that program.

[0] You might say that asbestos from 1 house would affect the whole neighborhood but that would be similar to how a smoker smoking in the streets would affect the neighborhood. Second-hand smoking and second-hand asbestos are bad but they're negligible compared to living with a smoker who smokes indoors with the windows closed or smoking yourself.