I doubt the CPR number alone will give you access to obtain credit and the like today, but you can absolutely go into a pharmacy and buy someone’s prescription medicine with just their CPR number, and you for sure can get access to a lot of data by calling various entities and providing your CPR number as proof of identity (but at least fewer now than used to be the case in the past).
If the CPR number was truly made public information, those cases would be much more obviously wrong. The fact that CPR numbers are de facto considered pseudo-secret makes things much worse.
In Norway I have to show photographic ID and my fødselsnummer.