This fails the moment one of your "clean line" contacts downloads a sketchy app that sells all their contacts, and sells an updated list as your appear in their recent calls list - meaning it's not a one-time thing, it's for a few weeks after every time you call them.
Scammers or spammers will never send an SMS with clarification that they wanted to call you.
As a parent, the "block all numbers!" approach has always seemed incredibly naive to me.
No police/fire/hospital emergency outcome is going to hinge upon someone else answering a phone call or text messsage.
When its someone legit I find that they actually go through the effort. I've had USPS explain who they were to my phone and I was able to answer them as a result.