I was going to ask how the web UIs possibly work if the location is indistinguishable, but I went to the Google Find Hub, and it appears you can't view the location for tags unless you enter your phone's pin code / pattern lock. This must either communicate with the phone or the keys are stored on Google's end.
EDIT: I turned my phone off, and I can still get the location of my keys... surely this doesn't mean the key is stored on their end? Wouldn't be very good E2EE in that case :)