This is a very useful precaution for banks, and for or calls that come from a family member's real phone number.
But scammers will just open with "I'm in trouble and my phone died" or "I'm in jail calling from a pay phone" and calling back won't do anything to help with that.
Also, given at least in America, our cell phone providers STILL haven't fully blocked caller ID spoofing (last I checked, they just add some tiny icon in the rare case that the CID is trustable, and I'd bet 99.9% of people don't even know that exists!) they can spoof the initial call as your number and many targets will probably mistakenly think the CID match is good enough to just skip it, especially in this "very urgent situation" with you being held at knifepoint by the corrupt third-world cops or whatever.
Not any old random number they give you.