There are times you need speaker-phone mode. My parents almost always turn on speaker-phone when they call me because they both want to be part of the conversation. I don't think they will ever take a plane or a bus trip in their lives so their speaker-phone isn't going to hurt anyone.

I need speakerphone when I'm home alone and attempting to be on a phone call while doing other things. Some of those calls are even about instructing me to look for something, so it necessitates me to be moving about. Speakerphone is an incredibly useful utility.

Don't take functionality away because of a few bad actors. That'd be like getting rid of drones because a few people are assholes.

Put rules in place to correct the bad behavior. Kicking them off planes seems fair.