At the end of the day, a man driving them home and knowing their home address while right there is a much more immediate and specific point of threat than "I'm being surveilled while I travel in a Google-controlled robotaxi".
Also, like it or hate it, but Google actually has a pretty decent track record with protecting their customers from their own employees. Gmail and gsuite content stays private, and there are enough protections in place to satisfy insurance companies, medical institutions, banks, and even governments. If a Waymo kidnapped or killed someone under the control of an overseas remote operator, it would be national news and congressional hearings; a person getting assaulted by a cab driver is nowhere near that level.
I just don't think "but Google/Waymo bad too" is a reason to poo poo someone feeling more comfortable sitting alone in a robotaxi than sitting alone in a cab or an uber.
Its not anymore of a threat than the taxi itself in case of Waymo. If you have already designated the driver as the threat actor then the Waymo is just as much a threat. In case of the driver the threat is visible. In a robotaxi the threat actor is invisible and multiple. You can be remotely locked in, driven away, suffocated, driven into water , driven in middle of a shooting.
You are just trading a visible threat with an invisible one.