This is done all the time, and is quite common, particularly when care is shared between multiple people or people are elderly
16% of all appointments are recorded by patients, and the number is increasing all the time. Only 3% covertly.
Basically all medical associations have issued guidance on this and the guidance is not "kick people out of the office"
I have never once had a provider become hostile or care. Nor has anyone I know. People record with phones, dial people in, etc.
This is across like 10+ states, tons of conditions, ages, and hundreds of people.
While you may have had a different experience, i dont' think i would go with "don't ever do this".
Maybe you have super-hostile relationships with your providers - nobody i know does.